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Appendix A. Methods 
This appendix describes the data and methodology used in the study. 


Data 

The School District of Philadelphia (SDP) provided data on teacher employment, teacher characteristics, student— 
teacher linkages, and school characteristics. The SDP Office of Talent Information Management provided data on 
teacher employment as of September 1, October 31, and April 1 of each school year; teacher characteristics by 
school year; teachers who left the district by school year; and scores on the teacher evaluation system by school 
year. The district’s Office of Research and Evaluation provided data on the grades and subjects that teachers 
taught by school year and data that enabled the study team to identify each student’s teachers by subject for 
research question 1. Finally, the Office of Research and Evaluation provided data on school characteristics, 
including student demographic characteristics, student proficiency level on state math and English language arts 
assessments, and school-level results from districtwide teacher surveys. A list of all data sources and variables 
used in the study, along with the years of data provided and used and the research questions informed by each 
set of variables, is in table A1. 


The study team created additional variables using data that SDP provided. These included teachers’ pension and 
retirement eligibility, teachers’ commute time from home to school for each school year, and the number of years 
a principal led a school by school year. Constructs were also created using scores on the teacher survey to describe 
teachers’ perceptions of school leadership, climate, student behavior, and the level of teacher autonomy, respect, 
and collaboration at the school. Additional details on how the study team created these variables appear after 
table A1. 


The study team also used data on school-level student-teacher ratios from the U.S. Department of Education’s 
Common Core of Data (U.S. Department of Education, 2017). These data were used in the analysis of research 
questions 3a and 3b. Data on each teacher’s class size were not available. The school-level student-teacher ratio 
denotes the number of students per teacher in a school. 


The years of data available differed by variable. To create consistency across the analyses, data from 2010/11 
through 2016/17 were used for analyses that included teacher characteristics other than teacher effectiveness, 
and data from 2014/15 through 2016/17 were used for analyses that included teacher evaluation ratings and 
effectiveness scores or school characteristics. For the timespan 2014/15 through 2016/17 the study team had 
both teachers’ Pennsylvania Value-Added Assessment System (PVAAS) scores and effectiveness scores from the 
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teacher evaluation system, as well as data on all school characteristics, including the percentage of economically 
disadvantaged students and teacher survey responses. 


Table A1. Data variables and sources for data provided by the School District of Philadelphia, 2010/11-— 


2017/18 


Source 

School District of Philadelphia Office of 
Talent Information Management, current 
active teachers (September 1, October 31, 
and April 1 time points) 

School District of Philadelphia Office of 
Talent Information Management, current 
active teachers as of October 31 


School District of Philadelphia Office of 
Talent Information Management, teachers 
who resigned, retired, or were terminated 
School District of Philadelphia Office of 
Research and Evaluation, data on courses 
and grade levels taught 


School District of Philadelphia Office of 
Talent Information Management, data on 
end-of-year teacher effectiveness 


School District of Philadelphia, Office of 
Research and Evaluation, student-teacher 
course linkages, grades 3-8 
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Years of data 
provided 
2010/11-2017/18 


2010/11-2017/18 


2010/11-2017/18 


2010/11-2017/18 


2013/14-2016/17 


2010/11-2017/18 


Years of data 
included in 
PEA 
2010/11-2016/17 


2010/11-2016/17 


2010/11-2016/17 


2010/11-2016/17 


2014/15-2016/17 


2014/15-2016/17 


AVETaF-] 0) (=k y={e | 

in analysis 
Teacher ID 
School ID 
Employment time 
point 
Teacher ID 
School ID 
School year 
Job title 
Gender 
Race/ethnicity 
Hire date 
Years since hire date 
Home address 
Attendance rate 
Teacher ID 
School year 
Date left the district 
Teacher ID 
School year 
Course name 
Course grade level 
taught 
Course subject area 
taught 
Teacher ID 
School year 
PVAAS score (0-3) 
Teacher 
effectiveness score 
(0-3) 
Teacher evaluation 
rating (failing, needs 
improvement, 
proficient, 
distinguished) 
Teacher ID 
Student ID 
School ID 
School year 
Course grade level 
Course subject area 
Student 
race/ethnicity 
Student 
economically 
disadvantaged 
status 


Research 
questions 
2, 3, 3a, 
and 3b 


All 
questions 


2, 3, 3a, 
and 3b 


All 
questions 


All 
questions 
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Years of data 


Years of data included in Variables used Research 
Source provided analysis in analysis questions 


School District of Philadelphia, Office of 2010/11-2017/18 2010/11-2016/17 e SchoollID All 
Research and Evaluation, longitudinal e School year questions 
school list e School address 
e School grade span 
e —- Principal name 

School District of Philadelphia Office of 2017/18 2017/18 e §=School ID 2, 3a, and 
Talent Information Management, e Employee ID 3b 
principals with start dates at school e Date principal 

started at the school 
School District of Philadelphia, Office of 2010/11-2017/18 2014/15-2016/17 e SchoollID 2, 3a, and 
Research and Evaluation, data on school e School year 3b 
enrollment and demographics e = Percentage of 

students by 

race/ethnicity 

2012/13-2017/18 2014/15-2016/17  e Percentage of 2 

economically 

disadvantaged 

students 
School District of Philadelphia, Office of 2010/11-2017/18 2014/15-2016/17 e SchoolID 2, 3a, and 
Research and Evaluation, results on the e School year 3b 
Pennsylvania System of School e = Percentage of 
Assessment and Keystone exams students by student 

proficiency level in 

math and English 

language arts 
School District of Philadelphia, Office of 2014/15-2017/18 2014/15-2016/17 e SchoolID 2, 3a, and 
Research and Evaluation, teacher e = School year 3b 
responses to district survey e = Percentage of 

teachers responding 

positively or 

negatively by survey 

question 


PVAAS is Pennsylvania Value-Added Assessment System. 
Source: Authors’ summary of data provided by the School District of Philadelphia for 2010/11—2017/18. 


Variables created by the study team 

Pension and retirement eligibility. The study team created variables of whether teachers were vested in the 
Pennsylvania Public School Employees’ Retirement System (PSERS) or eligible for retirement by using a 
combination of teachers’ hire date, age, and years employed in the district. Although PSERS eligibility is based on 
employment in public schools across the state, the study team had data only on teachers’ employment in SDP. 
Therefore, these variables may undercount the number of teachers who were vested or eligible for retirement. 
The variables were used in the analysis of research questions 3a and 3b because teachers’ PSERS eligibility could 
influence the likelihood that teachers would continue teaching in the district. 


Commute time. To estimate a teacher’s commute time between home and school, the study team used teachers’ 
home addresses from the Office of Talent Information Management’s data file on current active teachers and 
school addresses from the Office of Research and Evaluation’s longitudinal school list. First, the addresses were 
converted into latitude and longitude coordinates using the Esri ArcGIS geocoder. Then, using the Google Distance 
Matrix application programming interface, which includes traffic information, the driving time between each 
home and school address combination for a 7:30 a.m. arrival on a Tuesday was estimated. In some cases the same 
home and school address combination resulted in different commute times because of differences in the route 
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taken and variation in estimated traffic conditions. In these cases the study team averaged the estimated 
commute times. The maximum commute time was limited to two hours (affecting 0.2 percent of observations). 


Years principal led the school. To calculate the number of years a school principal led a particular school, the study 
team first used the Office of Talent Information Management’s data file of principals in the 2017/18 school year. 
This file included each school operating in 2017/18, the principal leading the school in 2017/18, and the date the 
principal began leading the school (which could have been before 2017/18). These data were used to calculate 
the number of years each principal led a particular school for all schools operating in 2017/18. However, the data 
file could not be used to calculate the years of principal leadership in all years from 2010/11 through 2016/17. 
Some schools closed during that period and therefore were not included in the 2017/18 data file. In addition, 
principals who left a school before 2017/18 were not included in the 2017/18 data file. For example, if a school’s 
principal in 2017/18 began at that school in 2014/15, the study team could calculate the years of principal 
leadership in 2014/15 (zero years), 2015/16 (one year), and 2016/17 (two years). However, the years of principal 
leadership at that school before 2014/15 could not be calculated. 


To address this limitation, the study team supplemented the data file of principals in the 2017/18 school year by 
using the principal names from the Office of Research and Evaluation’s longitudinal school list to look for changes 
in principals from 2001/02 through 2016/17. First, the study team created consistency across names by removing 
titles (for example, Mr., Ms., Dr.), hyphens, and other punctuation or language that would make it difficult to 
compare names across years. The team then noted when a principal’s first name, last name, or both changed from 
one year to the next. Years in which both the first and the last names changed were considered years in which the 
principal changed, and therefore the number of years of principal leadership started at zero. Years in which neither 
the first nor last name changed were considered years in which the principal stayed at the same school, and the 
years the principal led the school was increased by one. For schools with more than one principal, the principal 
was considered to be constant if at least one principal remained consistent across years. Schools could have more 
than one principal if one principal was on leave and another principal was filling in. In addition, the district might 
assign two or three principals to large schools. The study team reviewed cases in which the first name changed 
with no change in the last name, or the last name changed with no change in the first name. The majority of these 
cases were due to nicknames, misspellings, and compound names; therefore, the principal was assumed to be the 
same unless both the first and last name changed. 


Survey constructs. Results from the annual district teacher survey administered since 2014/15 were used to create 
six conceptually distinct constructs that described each teacher’s school environment. SDP provided survey results 
for schools with at least a 25 percent response rate or 25 teachers responding. The average response rate was 64 
percent and ranged from 25 percent to 100 percent. Eleven percent of schools in the study did not have survey 
results; however, it was not possible to determine whether this was because of low response rates at these schools 
or because the schools were excluded from the survey. The study team aligned the constructs with the topics SDP 
used to group survey questions (climate, instruction, leadership, professional capacity, and parent/guardian 
community ties). For each school year, constructs reflect the percentage of teachers in a school who responded 
positively to questions included in the construct. Construct values ranged from O percent to 100 percent and were 
calculated by dividing the total number of positive responses to each question (responses in the top two positive 
response categories on a four- or five-point scale) by the total number of responses for that question and then 
averaging the percentage of positive responses over all questions in the construct. No adjustments were made for 
nonresponse at the question level, meaning that teachers contributed data only for the questions they answered. 
Samples of the questions included in each construct are in table A2. 
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Table A2. Teacher survey constructs and sample questions, School District of Philadelphia, 2014/15-2016/17 


Survey construct 
School climate 
(43 questions) 


School leadership 
(11 questions) 


Teacher autonomy 
(16 questions) 


Teacher respect 
(8 questions) 


Student behavior 
(14 questions) 
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Sample questions 

How often are the following statements true about your classroom? (never, rarely, occasionally, or 

most or all of the time) 

-  Myclass is interrupted by announcements or messages from the office or colleagues 

-  Mystudents treat each other with respect 

How much do you agree or disagree with the following statements? (strongly disagree, disagree, 

agree, or strongly agree) 

- Curriculum, instruction, and learning materials are consistent among teachers in the same grade 
level at my school 

- Teacher morale is high at my school 

- Teachers at my school have high expectations for students 

To what extent do you consider each of the following factors a challenge to student learning in your 

schoo!? (a great challenge, a moderate challenge, a slight challenge, not a challenge) 

- Lack of high-quality professional development opportunities for teachers 

- Lack of school resources to provide extra help to students who need it 

- School crime/safety 

- Lack of support from parents and guardians 

The principal at this school (strongly disagree, disagree, agree, or strongly agree): 

- Communicates a clear mission for our school 

- Sets high standards for student learning 

- Sets clear expectations for teachers 

- Provides me with constructive feedback based on formal or informal observation(s) of my 
teaching 

- | Works to create a sense of community in this school 

- Promotes parent, guardian, and community involvement in the school 

How much control do you have over the following in your class? (none, a little, some, a great deal) 

- Determining course objectives 

- Choosing books and other instructional materials 

- Setting the pace for covering topics 

- Setting standards of behavior in my classroom 

- Choosing the teaching methods and strategies | use with my students 

- Choosing the evaluation and assessment activities | use in my class 

How much do you agree or disagree with the following statements? (strongly disagree, disagree, 

agree, or strongly agree) 

- lam encouraged to try new teaching approaches in my classrooms 

- lam encouraged to innovate to improve my teaching 

- lam willing to question others’ views on issues of teaching and learning 

- lam expected to continually learn and seek out new ideas 

- lam free to be creative in the teaching methods and strategies | use in the classroom 

To what extent do you feel respected by (not at all respected, a little respected, somewhat respected, 

respected a great deal): 

- District/charter administrators 

- Your principal 

- Teachers in your school 

-  Parents/guardians 

- Students 

How often are the following statements true about your classroom? (never, rarely, occasionally, most 

or all of the time) 

- My students complete their assigned work 

- | My students are motivated to learn 

- Students misbehave in my classroom 

- | My students ask me questions when they need help 

- | Mystudents work hard in my classes 
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Survey construct Sample questions 


Collaboration How often, if at all, do you collaborate with other teacher(s) or other school staff at your school 
(10 questions) about the following topics? (never or almost never, 1-4 times a year, 5—7 times a year, monthly or 

about monthly, weekly or about weekly, daily or almost daily) 

- Classroom management strategies/interventions 

- Identifying, developing, or revising curricular materials 

- Effective instructional strategies 

- The individual learning needs of students 

- Coordination of instruction across grade levels 


Source: Authors’ summary of questions from the School District of Philadelphia’s teacher survey. 


Sample 


The analytic sample was drawn from school personnel included in the Office of Talent Information Management’s 
data file on active teachers for each school year. The sample was limited to personnel with a “teacher” job title, 
which excluded personnel such as instructional specialists, academic coaches, and school counselors. To limit the 
sample to teachers working in kindergarten—grade 12, the sample excluded teachers with a preschool teacher title 
and teachers working in schools that did not serve kindergarten—grade 12. The data provided by the district did 
not include teachers in charter schools. 


Sample characteristics. Seventy-four percent of the teachers in the sample identified as female. Sixty-eight 
percent identified as White, 26 percent as Black, 3 percent as Hispanic, and 3 percent as another race/ethnicity 
(table A3). 


Table A3. Demographic characteristics of teachers in the School District of Philadelphia, overall and by school 
year, 2010/11-2016/17 (percent unless otherwise indicated) 


Overall 
Teacher 
demographic Sample 
characteristic size? Percent 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 
Gender 
Male 59,126 26.2 28.0 25.9 26.0 25.7 25.9 25.9 25.4 
Female 59,126 73.8 72.0 74.1 74.0 74.3 74.1 74.1 74.6 


Race/ethnicity 


Black 59,102 25.5 25.6 26.3 26.2 26.0 25.5 24.8 24.0 
Hispanic 59,102 2.9 2.7 2.7 2.7 3.0 3.1 3.1 3.1 
Other 59,102 3.4 3.4 3.3 3.4 3.5 3.4 3.5 3.7 
White 59,102 68.1 68.2 67.8 67.6 67.5 68.0 68.7 69.2 


a. The number of teacher-years represented in the analysis (teachers multiplied by their years employed in the district from 2010/11 through 2016/17). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11-2016/17. 


Roughly half of teachers taught math, English language arts, science, or social studies, and this remained 
consistent across school years (teachers could teach multiple subjects, so percentages do not sum to 100; table 
A4). A large majority (83 percent) of teachers were general education teachers; the rest (17 percent) were special 
education teachers. This breakdown also remained consistent across school years. By grade level 57 percent of 
teachers taught elementary school grades, 33 perent taught middle school grades, and 28 percent taught high 
school grades. The share of high school teachers declined during the study years, from 32 percent in 2010/11 to 
around 26 percent in later years. The number of schools at each grade level in the district changed because of a 
districtwide initiative in 2010/11 during which some schools were converted to charter schools and because of 
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school closures in 2011/12 and 2012/13, which might have changed the share of teachers at each grade level 
during the study years. 


Most (60 percent) teachers had regular attendance, but a substantial proportion (35 percent) were frequently 
absent, missing 10 or more days of teaching in a school year. Five percent of teachers missed 45 or more days in 
a school year; these teachers were included in the analyses as a separate group because it is likely that extenuating 
circumstances such as family or medical leave contributed to the high rate of absences. The percentage of teachers 
who were frequently absent increased over time, from 31 percent in 2010/11 to 38 percent in 2016/17 (see table 
A4). More than a quarter (28 percent) of teachers had a commute of less than 15 minutes, and nearly three- 
quarters (74 percent) of teachers had a commute of less than 30 minutes. 


Table A4. Professional characteristics of teachers in the School District of Philadelphia, overall and by school 
year, 2010/11-2016/17 (percent unless otherwise indicated) 


Overall 
Teacher professional Sample 
characteristic size? Percent 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 
Subject taught 
English language arts 49,650 50.7 53.8 52.7 50.6 50.5 50.0 48.8 48.4 
Math 49,650 54.2 58.7 55.8 53.8 54.0 53.2 52.6 51.4 
Science 49,650 47.5 49.4 48.2 48.0 48.0 47.2 46.4 45.1 
Social studies 49,650 51.4 55.4 52.5 51.4 51.3 50.9 49.8 48.4 


General education/special education 


General education 59,126 82.9 82.7 83.3 83.3 82.3 82.6 83.3 82.7 


Special education 59,126 17.1 17.3 16.7 16.7 17.7 17.4 16.7 17.3 


Grade level taught 


Elementary school grades 60,680 56.9 53.8 55.4 56.1 57.7 58.3 59.5 58.4 
(kindergarten—5) 

Middle school grades (6-8) 60,680 33.2 34.3 34.3 33.5 32.5 32.2 32.3 32.6 
High school grades (9-12) 60,680 28.1 32.1 28.6 28.5 27.1 26.4 26.0 26.8 
Attendance 

Regular (95 percent 59,104 60.3 64.3 61.7 59.8 59.0 59.9 58.3 57.9 
attendance or greater) 

Frequently absent (75-94 59,104 34.9 31.2 33.1 35.4 35.7 35.5 37.0 37.9 
percent attendance) 

Less than 75 percent 59,104 4.8 4.5 5.2 4.8 5.3 4.6 4.7 4.2 
attendance 


Commute time 


Shorter than 15 minutes 54,659 27.6 27.6 27.9 27.7 27.7 27.2 27.3 27.4 
15-29 minutes 54,659 46.8 47.1 46.8 46.6 46.1 46.4 46.7 47.5 
30-59 minutes 54,659 23.9 23.0 23.7 24.0 24.5 24.7 24.4 23.1 
60 minutes or longer 54,659 1.8 2.2 1.6 1.7 1.7 1.7 1.6 1.9 


Note: Teachers could teach multiple subjects or multiple grade levels; therefore, percentages for subject taught and grade level taught categories do not 
sum to 100. 

a. The number of teacher-years represented in the analysis (teachers multiplied by their years employed in the district from 2010/11 through 2016/17). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11—2016/17. 
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The percentage of teachers with four or fewer years of teaching in the district declined from 37 percent in 2010/11 
to 16 percent in 2014/15, then increased again to 26 percent in 2016/17 (table A5). The decline in the percentage 
of teachers new to the district between 2010/11 and 2014/15 may be due to restructuring and layoffs associated 
with the conversion of some schools to charter schools, school closures, and the 2012/13 budget shortfall. 


Table AS. Teachers’ years of teaching in the School District of Philadelphia, by school year, 2010/11-2016/17 
(percent unless otherwise indicated) 


hYol sleXe) BY(=t-1 8 Number of teachers 4 or fewer years 5-10 years 11-17 years More than 17 years 
2010/11 10,178 37.4 22.8 18.2 20.5 
2011/12 9,052 29.4 25.7 21.4 21.2 
2012/13 8,847 27.8 26.1 22.5 20.8 
2013/14 8,251 21.3 29.7 25.2 19.9 
2014/15 8,098 16.0 33.2 26.7 20.2 
2015/16 8,179 17.6 29.2 28.3 21.2 
2016/17 8,207 25.5 23.5 26.9 21.8 
All years (average)? 8,687 25.5 27.0 23.9 20.8 


a. The overall number of teachers across all years is an average of the number of teachers in the sample in each year. 

Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11—2016/17. 

Teachers receive one of four ratings in the educator evaluation system based on their underlying effectiveness 
score: failing (the lowest rating), needs improvement, proficient, or distinguished (the highest rating). Table A6 
shows the percentage of teachers in each evaluation rating category overall and by school year for 2014/15 
through 2016/17. 


Table A6. Distribution of teacher evaluation ratings and average scores in the School District of Philadelphia, 
overall and by school year, 2014/15-2016/17 (percent of teachers unless otherwise indicated) 


Teacher evaluation rating Overall 2014/15 2015/16 2016/17 
PVAAS score? 

Failing or needs improvement (0—1.49)? 14.1 7.8 6.9 47.8 
Proficient (1.5—2.49) 56.5 70.8 51.5 34.0 
Distinguished (2.5-3) 29.5 21.4 41.6 18.2 
Average PVAAS score (all categories) 2.1 2.1 2.3 1.4 
Sample size‘ 5,308 2,161 2,262 885 


Teacher effectiveness score 


Failing or needs improvement (0—1.49)? 4.1 4.8 29 4.7 
Proficient (1.5-2.49) 84.0 89.4 76.9 85.9 
Distinguished (2.5-3) 11.9 5.8 20.2 9.3 
Average effectiveness score (all categories) 2.1 2.0 2.2 2.0 
Sample size‘ 22,216 7,462 7,715 7,039 


PVAAS is Pennsylvania Value-Added Assessment System. 

Note: Percentages may not sum to 100 because of rounding. 

a.In school years 2014/15 and 2015/16 teachers were assigned a categorical PVAAS score of 0, 1, 2, or 3. In school year 2016/17 PVAAS scores were calculated 
as a rolling average of the three previous years’ PVAAS scores and therefore were a continuous measure. 

b. The bottom two evaluation rating categories (failing and needs improvement) were combined because of small sample sizes in the failing category. 

c. The number of teacher-years represented in the analysis (teachers multiplied by their years employed in the district from 2014/15 through 2016/17). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 
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Pennsylvania Value-Added Asssessment System (PVAAS) scores are one component of SDP’s educator evaluation 
system and provide an estimate of the academic growth of a teacher’s students based on students’ scores on the 
state assessment. PVAAS scores are available for teachers in tested grades and subjects (grades 4—8 and math and 
English language arts) beginning in 2014/15. In 2014/15 and 2015/16 teachers received a PVAAS score of 0, 1, 2, 
or 3. Beginning in 2016/17 PVAAS scores were calculated as a three-year rolling average based on scores from the 
previous three consecutive years, so only teachers with three years of PVAAS scores received scores in 2016/17. 
Because of the changes in how PVAAS scores were calculated and the composition of teachers who received 
PVAAS scores, analysis results presented in the main report do not include the 2016/17 PVAAS scores. 


A teacher’s effectiveness score is a summative measure that includes scores based on formal classroom 
observations, student learning objectives, PVAAS scores, and a school performance profile, which captures a 
school’s overall performance. A teacher’s effectiveness score is calculated based on the measures available for a 
teacher.’ All teachers receive an effectiveness score. However, beginning in 2016/17 teachers who were new to 
the district and had not taught for at least three years in another Pennsylvania district and teachers who had 
received an unsatisfactory rating in the previous year were placed in the peer assistance and review program and 
did not receive an effectiveness score. These teachers account for 3 percent of the sample and were not included 
in the analyses of teachers’ effectiveness scores. 


The distribution of teachers by effectiveness score rating varied from year to year, possibly because the district’s 
evaluation system had begun in 2013/14 and was still in its early years of implementation. Across years, though, 
a vast majority of teachers received a proficient rating (77-89 percent), and the smallest category was teachers 
with ratings of failing or needs improvement (3—5 percent; see table A6). Some 6—20 percent of teachers received 
the highest rating (distinguished) across school years. Average effectiveness scores were fairly consistent across 
years, ranging from 2.0 to 2.2. 


Analytic samples for research questions 1, 3a, and 3b. The analysis for research question 1 required data that 
linked individual students to teachers. Therefore, the sample for this analysis included only teachers with available 
student-teacher linkages in grades 3-8. The analysis was limited to grades 3-8 because the student-teacher 
linkages in these grades are likely to be more accurate because the linkages are used to calculate PVAAS scores. 
In addition, only teachers with effectiveness scores or PVAAS scores could be included in the analysis of differences 
based on teacher evaluation scores. Only teachers teaching subjects that are part of the Pennsylvania state 
assessments receive PVAAS scores, which further reduces the sample for the analysis of differences based on 
PVAAS scores,. 


The sample for the effective teaching gap (ETG) analysis for research question 1 also included students. The ETG 
estimates were based on 262,134 student-teacher observations for economically disadvantaged students and 
121,898 observations for non—economically disadvantaged students. Across all schools 73 percent of students 
identified as economically disadvantaged. The ETG estimates by student race/ethnicity included 56,448 
observations for White students, 190,934 observations for Black students, and 81,239 observations for Hispanic 
students. Schools with 10 or fewer students were excluded from the analysis. In 2017, 55 percent of students 
identified as Black, 19 percent as Hispanic, and 12 percent as White. In general, schools varied in their 
socioeconomic and racial/ethnic composition. Though most schools had a majority of students identified as Black 
or economically disadvantaged, there was sufficient variation within schools and between schools to conduct the 
ETG analysis (table A7). 


1 For example, for teachers with all measures available, observation scores account for 50 percent of their effectiveness score, student 
learning objectives account for 20 percent, PVAAS scores account for 15 percent, and the school performance profile accounts for 15 
percent. However, for teachers missing PVAAS scores, observations account for 50 percent, student learning objectives account for 35 
percent, and the school performance profile accounts for 15 percent (School District of Philadelphia, 2017). 
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Table A7. Selected school-level student demographic characteristics for the School District of Philadelphia, 
2014/15-2016/17 (percent of students) 


Identified as economically Identified as Identified as Identified as 
disadvantaged Black Hispanic White 


2014/ 2015/ 2016/ 2014/ 2015/ 2016/ 2014/ 2015/ 2016/ 2014/ 2015/ 2016/ 


Percentile se) 16 17 15 16 17 15 16 17 15 16 17 
Minimum 20.7 20.4 20.9 1.8 1.8 1.5 0.0 0.0 0.0 0.0 0.0 0.0 
25th percentile 60.6 64.5 68.2 28.4 24.2 25.0 2.6 2.4 3.1 1.0 1.1 1.0 
Median 70.2 72.3 76.7 56.6 59.1 55.8 9.5 75 7.8 2.7 2.2 2.9 
75th percentile 78.6 80.1 84.0 90.5 89.7 87.7 26.8 27.1 26.3 16.7 14.3 12.5 
Maximum 90.0 89.0 93.2 97.8 97.8 97.9 85.6 86.8 89.0 86.9 86.5 84.7 


Overall average 67.1 69.2 72.9 57.3 56.2 55.2 18.0 18.4 18.5 12.8 12.4 12.1 


Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


For research questions 3a and 3b only teachers with valid data on all the study variables were included in the 
sample. Because teacher survey data were not available until the 2014/15 school year, the analysis was limited to 
teachers employed from 2014/15 through 2016/17. Additionally, the sample included fewer novice teachers, 
because new teachers participated in the peer assistance and review process, which was mandatory for all new 
teachers who had not taught for at least three years in a public school district in the state. These teachers did not 
receive effectiveness scores in 2016/17, and therefore the sample of teachers for this year did not include teachers 
with less than three years of teaching experience in the state. Tables A8—A11 compare the characteristics of 
teachers included in the analytic sample for research questions 3a and 3b and the larger analytic sample included 
for research questions 2 and 3. 


For all research questions the study team imputed any missing data for the following variables: 


e School demographic characteristics. In 2015/16 and 2016/17, 54 percent of schools had student demographic 
data suppressed for at least one student subgroup because of the small number of students in a subgroup. To 
fill in the missing values, data were used for the closest year in which these data were available. 


e Teacher grade levels. Between 2014/15 and 2016/17, 16 percent of teachers were missing information about 
grade-level assignment for at least one year. The study team coded these teachers as teaching in 
kindergarten-grade 5, grades 6-8, or grades 9-12. The codes were based on the grade levels served by the 
school in which the teachers were employed in the school year for which they were missing a grade-level 
assignment. 


Comparison of analytic samples. Tables A8—A11 compare the characteristics and turnover rates of teachers in the 
larger sample used for research questions 2 and 3 with the more limited sample used for research questions 3a 
and 3b. Analyses for research questions 2 and 3 included all teachers employed from 2010/11 through 2016/17 
(except for analyses that examined school characteristics or teacher effectiveness, which were limited to 2014/15 
through 2016/17). For research questions 3a and 3b the sample was limited to teachers with data on all variables 
included in the analytic model. Therefore, the sample included only teachers employed from 2014/15 through 
2016/17, when complete data on school characteristics and teacher effectiveness were available. Table A8& 
compares the demographic and professional characteristics of the two samples, and tables A9 and A10 compare 
the evaluation ratings and years of teaching in the district for teachers in the samples. Table A11 compares the 
turnover rates of teachers in both samples. 
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The two samples were similar across characteristics, except that the limited sample used for research questions 
3a and 3b had fewer teachers with four or fewer years of teaching experience (17 percent) than the larger sample 
used for research questions 2 and 3 (26 percent; see table A10). This might be because more new teachers were 
placed in the peer assistance and review program in 2016/17 and therefore did not have effectiveness score data. 
This might also be because the percentage of teachers with four or fewer years of experience declined over the 
years of the study, with the highest percentages in the early years of 2010/11—2012/13 and the lowest 
percentages in 2014/15 and 2015/16 (see table AS). Likely as a result of having fewer new teachers, the 
percentage of teachers who left their school is lower among the sample for research questions 3a and 3b (16 
percent) than for the sample for the other research questions (25 percent; see table A11). This may also be 
because the data included in research questions 3a and 3b were limited to school years 2014/15 through 2016/17, 
when teacher survey data were available, whereas turnover rates in the district were highest in 2010/11 and 
2012/13, a period of district restructuring and budget shortfalls (see table B7). 


Table A8. Demographic and professional characteristics of teachers in the School District of Philadelphia, by 
analytic sample, 2010/11-2016/17 and 2014/15-2016/17 


Research questions 2 and 3 Research questions 3a and 3b 
(2010 /11-2016/17) (2014/15-2016/17) 
Teacher characteristic Percent Sample size? Percent Sample size? 
Gender 
Male 26.2 59,126 25.6 19,418 
Female 73.8 59,126 74.4 19,418 


Race/ethnicity 


Black 25.5 59,102 25.0 19,418 
Hispanic 2.9 59,102 3.2 19,418 
White 68.1 59,102 68.3 19,418 
Other race/ethnicity 3.4 59,102 3.5 19,418 
Subject taught 

English language arts 50.7 49,650 49.1 16,591 
Math 54.2 49,650 52.5 16,591 
Science 47.5 49,650 46.4 16,591 
Social studies 51.4 49,650 49.7 16,591 


Teaching assignment 


General education 82.9 59,126 82.8 19,418 


Special education 17.1 59,126 17.2 19,418 


Grade level taught 


Elementary school grades (kindergarten—grade 5) 56.9 60,680 59.0 19,418 
Middle school grades (6-8) 33.2 60,680 31.4 19,418 
High school grades (9-12) 28.1 60,680 26.4 19,418 
Attendance 

Regular (95 percent attendance or greater) 60.3 59,104 59.7 19,418 
Frequently absent (75-94 percent attendance) 34.9 59,104 36.6 19,418 
Less than 75 percent attendance 4.8 59,104 3.7 19,418 
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Research questions 2 and 3 Research questions 3a and 3b 


(2010 /11-2016/17) (2014/15-2016/17) 
Teacher characteristic Percent Sample size? Percent Sample size? 
Commute time 
Shorter than 15 minutes 27.6 54,659 27.5 19,418 
15-29 minutes 46.8 54,659 46.9 19,418 
30-59 minutes 23.9 54,659 24.2 19,418 
60 minutes or longer 1.8 54,659 1.4 19,418 


a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district during the years shown for each column). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11-2016/17. 


Table A9. Evaluation ratings among teachers in the School District of Philadelphia, by analytic sample, 
2014/15-2016/17 


Research questions 2 and 3 Research questions 3a and 3b 
Teacher evaluation rating Percent Sample size? Percent Sample size? 
PVAAS score? 
Failing or needs improvement (0—1.49)° 14.1 5,308 14.0 4,671 
Proficient (1.5-2.49) 56.5 5,308 56.5 4,671 
Distinguished (2.5—-3) 29.5 5,308 29.5 4,671 


Teacher effectiveness score 


Failing or needs improvement (0—1.49)° 4.1 22,216 4.2 19,418 
Proficient (1.5—2.49) 84.0 22,216 84.0 19,418 
Distinguished (2.5-3) 11.9 22,216 11.8 19,418 


PVAAS is Pennsylvania Value-Added Assessment System. 

a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district from 2014/15 through 2016/17). 

b.In school years 2014/15 and 2015/16 teachers were assigned a categorical PVAAS score of 0, 1, 2, or 3. In school year 2016/17 PVAAS scores were calculated 
as a rolling average of the three previous years’ PVAAS scores and therefore were a continuous measure. 

c. The bottom two evaluation rating categories (failing and needs improvement) were combined because of small sample sizes in the failing category. 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


Table A10. Years of teaching among teachers in the School District of Philadelphia, by analytic sample, 
2010/11-2016/17 and 2014/15-2016/17 


Research questions 2 and 3 Research questions 3a and 3b 


(2010 /11-2016/17) (2014/15-2016/17) 
Years of teaching in the district Percent Sample size? Percent Sample size? 
4 or fewer years 25.5 60,812 17.0 19,418 
5-10 years 27.0 60,812 30.8 19,418 
11-17 years 23.9 60,812 29.5 19,418 
More than 17 years 20.8 60,812 22.7 19,418 


a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district during the years shown for each column). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11—2016/17. 


Table A11. Teacher turnover in the School District of Philadelphia, by analytic sample and research question, 
2010/11-2016/17 and 2014/15—216/17 (percent of teachers unless otherwise indicated) 


Research questions 2and3 Research questions 3a and 3b 


(2010/11-2016/17) (2014/15-2016/17) 
Leaving their school for the full school year (Oct. to next Oct.) 24.8 16.3 
Leaving their school within the school year (Oct. to Apr.) 2.8 0.4 
Leaving their school between school years (Apr. to Oct.) 22.9 16.1 
Leaving their school but not the district for the full school year 16.8 11.4 
Leaving the district for the full school year 8.2 4.9 
Sample size (number)? 60,268 19,419 


a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district during the years shown for each column). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11—2016/17. 


Methodology 

Research question 1: How are teachers with different evaluation ratings distributed across schools in the district? 
Does student access to effective teachers vary between and within schools? The study team examined the 
distribution of teacher effectiveness scores and PVAAS scores across the district by calculating an intraclass 
correlation (ICC), which reports the degree of variation across schools as a percentage of the total variation in 
teacher effectiveness scores in the district. The ICC measures how similar teachers are within their schools and 
how similar teachers within each school are to teachers in different schools across the district. 


The study team also examined the distribution of effective of teachers by calculating the difference between the 
average effectiveness of teachers of economically disadvantaged, Black, and Hispanic students and the average 
effectiveness of teachers of non—economically disadvantaged and White students, a difference known as the 
effective teaching gap, or ETG (Isenberg et al., 2016). First, a district-level ETG was calculated as the average PVAAS 
score or effectiveness score of teachers of one group of students (economically disadvantaged, Black, or Hispanic) 
minus the average score of teachers of another group of students (non—economically disadvantaged or White). A 
simple regression was used for this calculation: 


Vix =at OSGjx + Cjk 


where Vj, is the PVAAS or effectiveness score of teacher j. Each teacher contributed two observations for a given 
subject and grade level: one for the subgroup (SG) of interest (economically disadvantaged, Black, or Hispanic), 
labeled k = 1 to represent the subgroup of interest (SG;; = 1), and one for the comparison group (non— 
economically disadvantaged or White), labeled k = 0 to represent the comparison group (SGjo = 0). A teacher’s 
score was the same for each observation (Vir = Vjo), and each observation was weighted by the total number of 
students of that subgroup. For example, a teacher who had 20 economically disadvantaged students and 10 non— 
economically disadvantaged students had weights of 20 and 10. The study team regressed Vj, on the binary 
variable SG; . The coefficient 6 measured the estimated difference in teacher scores between groups of students, 
with a positive value of 6 indicating that students in the subgroup of interest had teachers with higher scores, and 
a negative value of 6 indicating that students in the comparison group had teachers with higher scores on average. 


The study team then calculated a between-school ETG and a within-school ETG. The between-school ETG can 
indicate whether, for example, economically disadvantaged students attended schools with less effective teachers 
on average than did non—economically disadvantaged students. The within-school ETG can indicate whether 
economically disadvantaged students are assigned to less effective teachers within schools. The between-school 
ETG was calculated by averaging the effectiveness score or PVAAS score of all teachers in a given grade at each 
school in SDP and assigning each student the average score of teachers in their grade at their school. Each teacher 
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was then assigned the average of their students’ school grade-level scores. This average school grade-level score 
was used instead of each teacher’s individual score in the regression model (V;,). As with the district-level ETG, 
each teacher contributed two observations for a given subject and grade level (one for each student group), and 
a teacher’s score was the same for each observation. Each observation was weighted by the total number of 
students in each group. A positive 6 indicated that students in the subgroup of interest attended schools where 
teachers had higher scores, on average, and a negative 6 indicated that the students in the comparison group 
attended schools where teachers had higher scores. The within-school ETG was calculated as the difference 
between the overall district-level ETG based on teacher scores and the between-school ETG based on school 
scores. 


The between-school and within-school ETGs can reinforce or offset each other. For example, they reinforce each 
other if economically disadvantaged students attend schools that have teachers with lower average scores and 
are assigned teachers with lower average scores within these schools. They offset each other if economically 
disadvantaged students attend schools that have teachers with lower average scores but are assigned teachers 
with higher average scores within these schools. 


To allow for comparisons of ETGs across years and between PVAAS scores and effectiveness scores, the study 
team translated the ETGs into standard deviations of teacher PVAAS scores and effectiveness scores (see tables 
B3-B6 in appendix B). Standard deviations were calculated as the difference in the scores of teachers of two 
groups of students, divided by the standard deviation of all teachers’ scores in the subject, grade, and year. One 
standard deviation in teachers’ effectiveness scores was equal to 0.34 point, on average, across years, grades, and 
subjects (ranging from 0.30 point to 0.40 point). One standard deviation in PVAAS scores was equal to 0.69 point, 
on average (ranging from 0.61 point to 0.79 point). 


Teachers’ effectiveness scores combine multiple measures, including a school performance profile score and 
scores from classroom observations conducted by the school principal. Because the school performance profile 
score is the same for all teachers in a school and because school-specific observation practices may result in 
teachers’ observation scores being more similar within the same school than across different schools, the inclusion 
of these measures could magnify differences in teacher effectiveness scores between schools and minimize 
differences within schools. This potential bias is lower for teachers with PVAAS scores because their effectiveness 
scores also include the PVAAS scores, a measure based only on student achievement growth. 


To examine whether the inclusion of the school performance profile and observation scores biased the results for 
research question 1, the study team conducted a sensitivity analysis by limiting the analysis of effectiveness scores 
for research question 1 to teachers with PVAAS scores. The sensitivity analysis indicated that the results were not 
biased. The ICC results were of a similar magnitude, with 32 percent of the variation in effectiveness scores 
explained by differences between schools, on average. Between-school differences in average teacher 
effectiveness scores by student economic status and racial/ethnic identity were smaller across grade levels and 
subjects when the analysis was limited to teachers with PVAAS scores. The differences were about one-third the 
magnitude of differences when the analysis included all teachers with effectiveness scores. However, within- 
school differences were also smaller, indicating that differences in average teacher effectiveness scores continued 
to be driven by differences in average effectiveness scores between schools rather than by differences among the 
teachers assigned to students within schools. For analyses that included all teachers, within-school differences 
accounted for 15 percent of the total gap in average teacher effectiveness scores for grades 3—5 and 16 percent 
for grades 6-8 in math, and 21 percent for grades 3-5 and 30 percent for grades 6-8 in English language arts. For 
analyses that included only teachers with PVAAS scores, within-school differences accounted for less than 5 
percent of the total gap across grade levels and subjects. 


REL 2020-037 A-14 


The analyses for research question 1 used teacher effectiveness scores and PVAAS scores because they provide a 
continuous measure of teacher effectiveness. Using them assumes that differences in effectiveness scores are 
meaningful, even for teachers with the same evaluation rating (failing, needs improvement, proficient, and 
distinguished). Because a large proportion of teachers receive the same rating (proficient), it is reasonable and 
useful to distinguish between teachers within categories using the underlying effectiveness score. 


Research question 2: What proportions of teachers leave their school and leave the district each year, both during 
the school year and between school years? The study team calculated the turnover rate for each year as the 
percentage of teachers who left their school. This included teachers who changed schools, teachers who left 
teaching for a nonteaching position within the district, and teachers who left the district. In addition, the 
percentage of teachers who left their school but did not leave the district and the percentage of teachers who left 
the district were calculated separately. 


The study team compared teachers’ school assignments across employment time points to determine whether a 
teacher left a school, including changing schools or leaving teaching for a nonteaching position, during each study 
year. Teachers were classified as changing schools during the full school year if their school assignment changed 
between October 31 of one school year and October 31 of the following school year, and they were classified as 
changing schools within the school year if their school assignment changed between October 31 and April 1 within 
a school year. Teachers were classified as changing schools between school years if their school assignment 
changed between April 1 of one school year and October 31 of the following school year. For the same 
employment time points, teachers were classified as leaving teaching for a nonteaching position if they had a 
school assignment at one time point but did not have a school assignment at a future time point and were not 
recorded as having left SDP before April 1, 2017. 


To determine whether a teacher left the school district, the study team used data from the Office of Talent 
Information Management on teachers who resigned, retired, or were terminated in each school year. Teachers 
were classified as leaving the school district during the full school year if they were recorded as leaving between 
October 31 of that school year and October 31 of the following school year. A teacher who left between October 
31 and April 1 of a school year was classified as leaving the district within the school year. A teacher who left 
between April 1 of one school year and October 31 of the following school year was classified as leaving the district 
between school years. 


To calculate overall turnover rates, the study team pooled all years of the data and examined turnover rates by 
teacher characteristics and teacher evaluation ratings. To examine differences in turnover by school 
characteristics, the study team compared, for each characteristic, the average among schools in which teachers 
who left their school had taught before leaving with the average among schools in which teachers who stayed in 
their school had taught. These calculations were repeated for schools in which teachers who left the district had 
taught before leaving and schools in which teachers who stayed in the district had taught. 


Research question 3: How does teaching experience relate to whether teachers leave their school or leave the 
district? For research question 3 the study team used a survival analysis to examine the likelihood that teachers 
would leave their school during the years in which they were observed teaching in that school or would leave the 
district over the course of their career in the district. The survival analysis examined the amount of time until a 
teacher left a school or left the district and calculated the rate at which teachers left at any given point during 
their time in a school or in the district. The study reports the failure function, which is the cumulative probability 
that teachers would leave their school or leave the district over time, and the hazard rate for each year, which is 
the rate at which teachers left their school or left the district at each time point. 


Time was measured in days, beginning with a teacher’s hire date in the district and ending at the date the teacher 
left the district or the last date at which the teacher was observed in the data. Teachers’ employment status was 
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observed at three points in each school year: September 1, October 31, and April 1. The first possible date when 
a teacher could have been observed was September 1, 2010, and the last possible date was April 1, 2017. To 
account for the fact that the study could not observe teachers who left the district before September 1, 2010, 
teachers contributed to the analysis only for the years of teaching in which they were observed. For example, a 
teacher who was hired in September 2005 would be included in the analysis starting at five years of teaching 
(September 2010), which is the point in the teacher’s career at which the study could observe whether the teacher 
left the school or left the district. If this teacher left in August 2015 (at 10 years of teaching), the teacher would 
be included in calculating the probability of a teacher leaving between 5 and 10 years of teaching. This teacher 
would not be included in calculating the probability of a teacher leaving between 1 and 4 years of teaching because 
the teacher was not observed during those years. 


The study team conducted two survival analyses: an analysis of the probability that a teacher would leave a school 
over the years teaching in a school and an analysis of the probability that a teachers would leave the district over 
the years employed in the district. Each analysis is summarized below. The study team decided not to analyze the 
factors related to a teacher’s decision to move within the district, excluding teachers who left the district. When 
a teacher leaves a school, it can disrupt the school community and requires spending resources to hire a 
replacement teacher, regardless of whether the teacher leaves for another school in the district or leaves the 
district. 


Analysis of teachers who leave their school. As described in the methodology for research question 2, the study 
team compared teachers’ school assignments across time points to determine whether a teacher left a school 
during each study year. The date of a teacher’s last time point employed at a school was used as the date the 
teacher left the school. For example, if a teacher was employed in School A on April 1, 2015, and employed at 
School B on September 1, 2015, that teacher was considered to have left School A on April 1, 2015. For teachers 
who left the district, the study team used the date the teacher left—obtained from the Office of Talent 
Information Management’s data file on teachers who resigned, retired, or were terminated in each school year— 
as the date that the teacher left a school. Teachers with a final time point employed in a school that was before 
April 1, 2017, but who were not recorded as leaving the district were also considered to have left their school 
because these teachers might have moved into a nonteaching position within the district. For these teachers the 
last time point employed at a school was used as the date the teacher left the school. 


To account for the fact that teachers could change schools multiple times during their careers, time was measured 
beginning with a teacher’s hire date in the district for the first school in which the teacher was employed and, for 
subsequent schools, at the date of the first time point at which the teacher was employed in the school. For 
example, if a teacher was first observed at School A in September 2010 and then moved to School B in April 2013, 
time was measured from the teacher’s hire date for the teacher’s tenure in School A and then measured again 
from April 2013 for the teacher’s tenure in School B. This approach allowed the analysis to include multiple school 
moves and examine teacher turnover by the amount of time a teacher was employed in a school. The study team 
chose this approach because it is likely most useful for school principals to understand the probability that a 
teacher will leave the school within the context of how long the teacher has taught at that school, rather than 
how long the teacher has been employed in the district. Thirty-six percent of teachers changed schools at least 
once during the study period. This analysis included 174,373 teacher employment observations representing 
21,999 teacher—school combinations and 14,578 teachers. 


Analysis of teachers who leave the district. As described above for research question 2, the study team used data 
from the Office of Talent Information Management on teachers who resigned, retired, or were terminated in each 
school year to determine whether a teacher left the district. The date the teacher left the district was the date 
recorded in the data file. The survival analysis of teachers leaving the district included 188,731 teacher 
employment observations representing 14,967 teachers. 
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Research question 3a: What other teacher and school characteristics are related to teachers leaving their school 
or leaving the district? For research questions 3a and 3b the study team used a Cox proportional hazards model, 
which analyzes the factors associated with the occurrence and timing of events (Kalbfleisch & Prentice, 2002; 
Singer & Willett, 2003). As with the survival analysis, the study team used the Cox proportional hazards model to 
examine which teacher and school characteristics were related to two types of events: teachers leaving a school 
and teachers leaving the district. Only teachers with complete data across all time points were included in the 
hazards models, which limited the sample to school years 2014/15 through 2016/17; sample sizes are provided 
below in the description of each hazards model. 


For both hazards models the study team examined correlations between independent variables to assess potential 
collinearity before fitting each hazards model. After fitting each model, the study team tested the proportional 
hazards assumption that the change in risk with each covariate is constant over time. This assumption might be 
violated if, for example, a variable increases a teacher’s risk of leaving in the first five years of teaching but then 
has no effect on leaving later in a teacher’s tenure. The proportional hazards assumption was tested using 
Schoenfeld residuals, which calculate the difference between the value of a variable at the time a teacher leaves 
and the average value of that variable among all teachers who have not left by that time point (Cleves et al., 2016; 
Singer & Willett, 2003). Schoenfeld residuals detect a violation of the proportional hazards assumption by 
examining whether differences increase or decrease over time (indicating a violation) or stay the same (indicating 
no violation). In the example above a larger difference would exist between the value of the variable for teachers 
who leave and the average value for teachers who stay during the first five years of teaching than in subsequent 
years, indicating a violation of the assumption. 


Hazards model specifications for teachers who left a school. The date of the last time point at which a teacher was 
employed at a school, or the date the teacher left the district (if the teacher left the district), was used as the date 
the teacher left a school. Time was measured in days from the hire date in the district for the first school at which 
a teacher was observed until the teacher left the school or left the district. For teachers who changed schools 
during the study period, time began again, starting with day zero, at the date of the first time point at which a 
teacher was observed at each subsequent school. 


The model accounted for the date at which teachers were first observed in the data and used robust standard 
errors clustered at the teacher level. The independent variables used in the hazards model are in table B15 in 
appendix B and include a variable accounting for the number of times a teacher had previously changed schools. 
Forty-three percent of teachers changed schools at least once during the study period. No variables violated the 
proportional hazards assumption. The hazards model for teachers leaving a school included 48,052 teacher- 
employment observations representing 9,219 teacher—school combinations and 8,138 teachers. 


Hazards model specifications for teachers leaving the district. Time was measured in days from the hire date in 
the district until the date the teacher left the district. The model also specified the date at which teachers were 
first observed in the data. Robust standard errors were clustered at the teacher level. The independent variables 
used in the hazards model are in table B15 in appendix B. The hazards model for teachers leaving the district 
included 50,147 teacher employment observations representing 8,147 teachers. 


For this model the retirement eligibility variable violated the proportional hazards assumption, meaning that its 
influence over teachers’ probability of leaving changed over time. This is not surprising. Teachers typically become 
eligible for retirement near the end of their career, so retirement eligibility would have no effect early in a 
teacher’s career. To adjust for this, the study team estimated a model interacting retirement eligibility with time. 
This model provided a slightly better fit; however, the report uses results from the model without the time- 
dependent effects for ease of explanation. The results with time-dependent effects are in table B17 in appendix 
B. Results from the model with time-dependent effects are similar to results from the model without them. The 
same characteristics are significantly related to the probability of teachers leaving the district in both models. 
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Coefficients in both models did not differ for 8 of the characteristics and were similar for the remaining 13 
characteristics. 


Research question 3b: Does the relationship between teacher and school characteristics and turnover differ 
between effective teachers and ineffective teachers? To examine whether the relationship between teacher and 
school characteristics and teacher turnover differed, the study team interacted a variable indicating whether a 
teacher was rated proficient or distinguished with each of the independent variables in the hazards model. The 
team then examined whether the relationship between each variable and teachers’ likelihood of leaving was both 
significantly and substantively different between teachers with high or low evaluation ratings. 


Limitations 

Data availability. The study team’s analysis of characteristics related to turnover was limited to characteristics 
measured in the data the district provided. Although data captured several important teacher and school factors, 
other unmeasured factors could have influenced teacher turnover, such as teachers’ induction or mentoring 
programs, teachers’ preparation, and teachers’ prior experience with similar student populations. 


Constructs from the teacher survey data. Teacher survey results may not be representative at the school level 
because of low response rates overall and at some schools (see description of survey response rates above). 
Response rates were not related to schools’ academic performance or student demographics. Smaller schools had 
higher response rates, so survey results may be more representative of teachers in smaller schools than teachers 
in larger schools. Higher response rates were related to more positive teacher responses on the survey, indicating 
that survey results may be more representative of teachers at schools with a more positive school climate. 
However, response rates might provide a measure of the level of teacher engagement at a school, suggesting that 
low response rates could reflect a less positive school climate. While the teacher survey results might differ if 
response rates were higher, the survey results still provide a useful and unique measure of teachers’ perspectives 
about their school that can inform SDP’s policies. 


In addition, the study team did not assess the reliability and validity of the survey constructs used in the analysis. 
However, these constructs are closely aligned with survey constructs that SDP created and assessed for reliability 
and validity (School District of Philadelphia, 2018). 


The construction of teacher effectiveness and PVAAS scores. Teacher effectiveness scores in the district combine 
multiple measures and may include scores from principal observations of teacher practice, PVAAS scores, scores 
based on student learning objectives, and a school performance profile intended to capture a school’s overall 
performance. Each of these measures could introduce bias that would affect the analysis of research question 1 
and potentially overstate differences in teacher effectiveness between schools and between groups of students. 
A school receives one performance profile score that is applied to all teachers in that school and is based on a 
combination of academic and nonacademic measures, including student scores on state assessments, graduation 
or promotion rates, and attendance rates. The inclusion of this measure and observation scores that might be 
influenced by school-specific practices could bias the results for research question 1 by magnifying differences in 
average effectiveness scores between schools and minimizing differences within schools. As described in the 
methodology section, a sensitivity analysis indicated that the inclusion of school performance profile and 
observation scores did not bias the results for research question 1. Other research also indicates that observation 
measures may be biased, with teachers of economically disadvantaged students and students of color more likely 
to receive lower observation scores than teachers of non—economically disadvantaged students or White 
students (Campbell & Ronfeldt, 2018; Steinberg & Garrett, 2016; Whitehurst et al., 2014). This bias could lead to 
overestimating differences in the quality of teachers of different groups of students in the analysis of research 
question 1. 
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Similarly, PVAAS scores could overestimate differences in teacher quality because they do not include controls for 
some student characteristics, such as special education status and English learner status, and do not account for 
the effects of a student’s classmates on the achievement of that student in a classroom or for the effects of the 
clustering of similar types of students within schools and classrooms (Kane, 2017; McCaffrey & Buzick, 2014; 
Isenberg et al., 2016). In addition, PVAAS scores are calculated as a point estimate divided by the standard error 
of the measure. Teachers of economically disadvantaged students might have less precise scores, and therefore 
larger standard errors, because it is more difficult to accurately predict the future achievement of these students 
(Hermann et al., 2016). Therefore, teachers of economically disadvantaged students might be less likely to receive 
very high or very low PVAAS scores. 


To address potential bias and limitations in the effectiveness scores and PVAAS scores, the report presents results 
using both measures of teacher effectiveness and discusses similarities and differences in the findings. 


Applying findings to all district teachers. The analysis of the combined influence of teacher and school 
characteristics on the likelihood of teacher turnover could include only teachers with complete data across all 
school years. This limited the years included in the analysis to 2014/15 through 2016/17, meaning that the findings 
apply only to teachers employed in more recent years. Most of the characteristics of teachers included in this 
analysis were similar to the characteristics of teachers included in the analysis of other research questions, except 
that these teachers were slightly more experienced than the overall teacher population. Thus, findings may be 
more applicable to more experienced teachers (see tables A8—A11). 


Analyses using PVAAS score data cannot generalize to all district teachers because only teachers who taught math 
and English language arts in grades 4-8 received PVAAS scores. 


Finally, the district experienced higher teacher turnover in 2010/11, because of a districtwide initiative converting 
schools to charter schools, and in 2012/13, because a budget shortfall that led to staff layoffs and school closings. 
Therefore, analyses that include these school years may not reflect teacher turnover in years without these 
events. 


Macroeconomic influences on the district’s workforce. The study included school years 2010/11 through 2016/17, 
which includes the 2008 recession and recovery period. The economic environment varied substantially over the 
study time period. The unemployment rate in the Philadelphia area declined from 8.4 percent in September 2010 
to 4.4 percent in April 2017 (U.S. Department of Labor, 2019). Changing economic conditions could influence 
teacher turnover, including the district’s ability to hire new teachers and retain existing teachers. The study does 
not take into account the potential influence of the economy on teacher turnover; therefore, findings might differ 
in another economic environment. 
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Appendix B. Supporting analyses 
This appendix includes tables that support the study findings presented in the main report. 


Research question 1: How are teachers with different evaluation ratings distributed across schools in 
the district? Does student access to effective teachers vary between and within schools? 

This section presents results for research question 1 by school year and subject area for findings summarized in 
the main report and presents findings from additional analyses not discussed in the main report. 


The relationship between teacher experience and evaluation rating. About 9 percent of teachers with 4 or fewer 
years of experience in the district were rated distinguished based on their effectiveness score, while 14 percent 
of teachers with more than 17 years of experience were rated distinguished (table B1). Teachers with four or 
fewer years of experience were about as likely as teachers with more experience to be rated failing or needs 
improvement (5 percent compared with 4 percent). The results were similar for Pennsylvania Value-Added 
Assessment System (PVAAS) scores: teachers with four or fewer years of experience were less likely than teachers 
with more experience to be rated distinguished. 


Table B1. Evaluation ratings among teachers in the School District of Philadelphia, by effectiveness measure 
and by years of teaching experience, 2014/15-2016/17 


Overall Years of teaching experience (percent of teachers) 
Effectiveness measure and evaluation 4 or fewer More than 
rating Percent Sample size? VEEL 5-10 years 11-17 years WANES 
Teacher effectiveness score 
Failing or needs improvement (0—1.49)? 4.1 22,216 5.1 3.9 4.0 3.9 
Proficient (1.5-2.49) 84.0 22,216 86.1 84.0 83.9 81.9 
Distinguished (2.5-3) 11.9 22,216 8.8 12.1 12.0 14.2 
Average effectiveness score 2.06 22,216 2.02 2.07 2.06 2.09 
Pennsylvania Value-Added Assessment System (PVAAS) score 
Failing or needs improvement (0—1.49) 7.2 4,423 12.8 6.4 5.3 6.1 
Proficient (1.5-2.49) 60.9 4,423 61.7 61.9 61.4 58.1 
Distinguished (2.5-3) 31.9 4,423 25.5 31.7 33.3 35.8 
Average PVAAS score 2.21 4,423 2.05 2.22 2.26 2.26 


Note: Observations are by teacher-year. PVAAS data include school years 2014/15 and 2015/16. 

a. The number of teacher-years included in the analysis (teachers multiplied by their years employed in the district from 2010/11 through 2016/17). 
b. The bottom two evaluation rating categories (failing and needs improvement) were combined because of small sample sizes in the failing category. 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


The distribution of effective teachers across the district. Table B2 presents findings from the analysis of the 
distributions of teacher effectiveness scores and PVAAS scores across the district. Because results are consistent 
across school years and grade levels, the main report presents a weighted average across all years and grades. 
This section provides detailed findings by school year and grade level. 


As described in the main report and appendix A, the study team calculated a measure of how similar teachers 
were within the same school and how similar each school’s average teacher was to the average teacher at other 
schools. This measure, the intraclass correlation, reports the percentage of the total range of teacher effectiveness 
scores or PVAAS scores that can be explained by differences between each school’s average teacher. An intraclass 
correlation of 0 percent indicates that teacher effectiveness is perfectly distributed across schools and that each 
school has the same average teacher effectiveness score or PVAAS score. An intraclass correlation of 100 percent 
indicates that each school has a different average teacher effectiveness score or PVAAS score and that all teachers 
within a school have the same score. For example, for elementary school teachers 8-18 percent of the variation 
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in teachers’ PVAAS scores was explained by differences between schools. By comparison, 22—34 percent of the 
variation in teachers’ effectiveness scores was explained by differences between schools. This means that there 
were larger differences between schools in average teacher effectiveness scores than in average PVAAS scores. 


Table B2. Between-school differences in the percentage of total variation in teacher evaluation scores in the 
School District of Phiadelphia, by grade level and school year, 2014/15-2016/17 


Teacher effectiveness score AVPAVANIEY 0) =) 


Intraclass correlation Intraclass correlation 


coefficient Number of coefficient Number of 
(95% confidence interval) teachers (95% confidence interval) teachers 


Elementary school teachers (kindergarten—grade 5) 


2014/15 0.28 3,371 0.18 1,208 
(0.23, 0.34) (0.12, 0.24) 

2015/16 0.22 3,575 0.10 1,320 
(0.17, 0.27) (0.06, 0.15) 

2016/17 0.34 3,219 0.08 425 
(0.28, 0.40) (0.00, 0.19) 


Middle school teachers (grades 6-8) 


2014/15 0.34 1,614 0.24 1,300 
(0.27, 0.42) (0.17, 0.30) 

2015/16 0.23 1,661 0.16 1,364 
(0.17, 0.30) (0.11, 0.22) 

2016/17 0.39 1,489 0.20 490 
(0.31, 0.47) (0.10, 0.30) 


High school teachers (grades 9-12) 


2014/15 0.36 1,851 na na 
(0.24, 0.47) 

2015/16 0.37 1,862 na na 
(0.26, 0.49) 

2016/17 0.36 1,775 na na 
(0.25, 0.47) 


PVAAS is Pennsylvania Value-Added Assessment System. ICC is intraclass correlation coefficient. n.a. is not applicable. 

Note: PVAAS scores are available only for teachers in grades 4—8; therefore, there is no PVAAS intraclass correlation coefficient for high school teachers, and 
the PVAAS intraclass correlation coefficient for elementary school teachers includes only teachers in grades 4 and 5. 

Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


Differences in the average effectiveness of teachers of economically disadvantaged, Black, and Hispanic students 
and teachers of non—economically disadvantaged and White students. Tables B3—B6 present findings from the 
effective teaching gap (ETG) analysis by school year, grade level, and measure of teacher effectiveness for three 
types of ETGs: 


e Between-school ETG, which indicates whether one group of students attends schools with less effective 
teachers, on average. 


e Within-school ETG, which indicates whether one group of students is assigned to less effective teachers within 
schools. 
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e _District-level ETG, which describes the overall difference between the average effectiveness of teachers of 
one group of students (economically disadvantaged, Black, or Hispanic) compared with teachers of another 
group (non—economically disadvantaged or White). 


To enable comparisons across years, grade levels, and measures of teacher effectiveness, results are presented in 
standard deviations of teachers’ scores (calculated as the difference in the scores of teachers of two groups of 
students, divided by the standard deviation of all teachers’ scores). A negative ETG indicates that non- 
economically disadvantaged or White students had teachers with higher scores on average than economically 
disadvantaged, Black, or Hispanic students. A positive ETG indicates that economically disadvantaged, Black, or 
Hispanic students had teachers with higher scores. 
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Table B3. Effective teaching gap in math in grades 3-5 in the School District of Philadelphia, by effectiveness 
measure, student group, and school year, 2014/15-2016/17 


Effective teaching gap 
(standard deviation) 


Effectiveness measure, student Number of Number of Between Within ot ¢-11 | 
group, and school year teachers schools schools Yel store) ky district 


Gap in teacher effectiveness scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 1,308 158 -0.22** —0.04 —0.26** 
2015/16 1,292 157 —0.13" 0.01 -0.13* 
2016/17 1,168 157 -0.41'* 0.07 —0.33"* 


Black students relative to White students 


2014/15 1,282 158 -0.56'* -0.07 —0.63** 
2015/16 1,254 157 -0.47** 0.09* 0.39" 
2016/17 1,136 156 -0.99"* 0.23** —0.76"* 


Hispanic students relative to White students 


2014/15 1,054 153 -0.62** 0.03 —0.60** 
2015/16 1,049 155 -0.35'* 0.04 -0.30"t 
2016/17 969 153 -0.56"* 0.05 -0.51"* 


Gap in teacher PVAAS scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 673 158 -0.07 -0.01 —0.08 
2015/16 636 157 —0.06 0.01 —0.05 
2016/17 344 156 -0.14" -0.11 -0.24" 


Black students relative to White students 


2014/15 666 158 -0.15* —0.03 -0.18" 
2015/16 623 157 -0.28"* 0.08 -0.20* 
2016/17 341 155 -0.20" 0.22" —0.42'* 


Hispanic students relative to White students 


2014/15 551 150 -0.38tt 0.07 —0.30** 
2015/16 546 152 —0.22t 0.03 -0.19t 
2016/17 297 152 —0.08 —0.20+ —0.28** 


PVAAS is Pennsylvania Value-Added Assessment System. 

t Effective teaching gap (ETG) is more than half the difference in scores between teachers in their first four years of teaching and teachers with five or more 
years of teaching. 

tt ETG is larger than the difference in scores between teachers in their first four years of teaching and teachers with five or more years of teaching (0.17 
standard deviation for effectiveness scores and 0.26 standard deviation for PVAAS scores). 

Note: PVAAS scores are available only for teachers in grades 4 and 5. 

Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15-2016/17. 


Table B4. Effective teaching gap in English language arts in grades 3—5 in the School District of Philadelphia, by 
effectiveness measure, student group, and school year, 2014/15-2016/17 


Effective teaching gap 
(standard deviation) 


Effectiveness measure, student Number of Number of Between Within Overall 
group, and school year teachers schools schools schools district 


Gap in teacher effectiveness scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 1,423 159 -0,24't 0.01 0.24" 
2015/16 1,423 158 -0.18** 0.03 -0.15* 
2016/17 1,279 156 -0.34** 0.06 —0.28"t 


Black students relative to White students 


2014/15 1,381 159 -0.55*t -0.01 —0.56't 
2015/16 1,362 157 -0.51** 0.19** —0.33't 
2016/17 1,231 155 -0.89** 0.25** —0.64** 


Hispanic students relative to White students 


2014/15 1,153 154 -0.61** 0.00 -0.61** 
2015/16 1,160 156 -0.44tt 0.10* -0.34't 
2016/17 1,073 153 -0.55** 0.07 —0.48"t 


Gap in teacher PVAAS scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 709 158 -0.05 -0.01 —0.06 
2015/16 705 158 -0.10 0.02 —0.07 
2016/17 359 156 -0.147 -0.10 -0.23" 


Black students relative to White students 


2014/15 698 158 0.03 -0.11 —0.08 
2015/16 687 157 -0,34tt 0.16" -0.18* 
2016/17 357 155 -0.217 —0.25+ -0.46'* 


Hispanic students relative to White students 


2014/15 583 150 -0,34tt 0.14* -0.21* 
2015/16 597 153 -0.38"* 0.15" -0.23" 
2016/17 307 152 -0.197 -0.17* —0.35't 


PVAAS is Pennsylvania Value-Added Assessment System. 

t Effective teaching gap (ETG) is more than half the difference in scores between teachers in their first four years of teaching and teachers with five or more 
years of teaching. 

+t ETG is larger than the difference in scores between teachers in their first four years of teaching and teachers with five or more years of teaching (0.17 
standard deviation for effectiveness scores and 0.26 standard deviation for PVAAS scores). 

Note: PVAAS scores are available only for teachers in grades 4 and 5. 

Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


Table B5. Effective teaching gap in math in grades 6-8 in the School District of Philadelphia, by effectiveness 
measure, student group, and school year, 2014/15-2016/17 


Effective teaching gap 
(standard deviation) 


Effectiveness measure, student Number of Number of Between Within o)V(Te-11 | 
group, and school year teachers schools schools schools district 


Gap in teacher effectiveness scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 631 146 -0.22** —0.08 -0.30** 
2015/16 595 141 -0.24't 0.02 -0.21'* 
2016/17 512 142 -0.47** 0.09 -0.38'* 


Black students relative to White students 


2014/15 609 144 -0.72't -0.05 -0.77** 
2015/16 585 140 -0.50"* 0.06 -0.43"* 
2016/17 494 140 -0.79** 0.117 -0.69'* 


Hispanic students relative to White students 


2014/15 481 126 -0.42** —0.05 -0.48'* 
2015/16 474 124 -0.42** 0.05 -0.38'* 
2016/17 425 130 -0.67** 0.137 -0.55** 


Gap in teacher PVAAS scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 441 146 -0.11 —0.02 -0.13" 
2015/16 409 141 -0.23" 0.12 -0.11 
2016/17 236 142 -0.24" —0.03 -0.27"t 


Black students relative to White students 


2014/15 436 144 -0.41't -0.07 -0.48't 
2015/16 404 140 -0.53** 0.347* —0.18" 
2016/17 234 140 -0.48** —0.04 -0.52** 


Hispanic students relative to White students 


2014/15 356 126 -0.10 -0.15* -0.25+ 
2015/16 355 124 -0.56"* 0.26** -0.30"* 
2016/17 217 130 —0.38** 0.02 -0.36'* 


PVAAS is Pennsylvania Value-Added Assessment System. 

t Effective teaching gap (ETG) is more than half the difference in scores between teachers in their first four years of teaching and teachers with five or more 
years of teaching. 

tt ETG is larger than the difference in scores between teachers in their first four years of teaching and teachers with five or more years of teaching (0.20 
standard deviation for effectiveness scores and 0.26 standard deviation for PVAAS scores). 

Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15-2016/17. 


Table B6. Effective teaching gap in English language arts in grades 6-8 in the School District of Philadelphia, by 
effectiveness measure, student group, and school year, 2014/15-2016/17 


Effective teaching gap 
(standard deviation) 


Effectiveness measure, student Number of Number of Between Within oF e-11 | 
group, and school year teachers schools schools FXolsYore) ky district 


Gap in teacher effectiveness scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 720 144 -0.38't 0.02 -0.36"" 
2015/16 676 142 -0.36'" 0.09 -0.27"" 
2016/17 595 141 -0.53"" 0.18" -0.34"t 
Black students relative to White students 

2014/15 686 143 -0.75"" 0.10° -0.65"t 
2015/16 648 141 -0.62** 0.13" -0.48*t 
2016/17 570 138 -1.20'* 0.53"t -0.67"t 
Hispanic students relative to White students 

2014/15 557 124 -0.63'" 0.07 -0.57"" 
2015/16 530 125 -0.60'* 0.01 -0.59'* 
2016/17 498 130 -0.82"* 0.25" -0.57"" 


Gap in teacher PVAAS scores 


Economically disadvantaged students relative to non—economically disadvantaged students 


2014/15 482 144 -0.22" 0.10 -0.13" 
2015/16 449 142 -0.28't 0.09 -0.19" 
2016/17 248 141 -0.16" —0.06 -0.22" 
Black students relative to White students 

2014/15 474 143 -0.56"t 0.24" -0.317" 
2015/16 438 141 -0.50'T 0.17" -0.32"* 
2016/17 245 138 -0.41"t —0.07 -0.48"t 
Hispanic students relative to White students 

2014/15 395 124 -0.45"t 0.24" -0.21" 
2015/16 379 125 -0.51"* -0.01 -0.52"* 
2016/17 223 130 -0.36"" 0.02 -0.34"" 


PVAAS is Pennsylvania Value-Added Assessment System. 

t Effective teaching gap (ETG) is more than half the difference in scores between teachers in their first four years of teaching and teachers with five or more 
years of teaching. 

tt ETG is larger than the difference in scores between teachers in their first four years of teaching and teachers with five or more years of teaching (0.20 
standard deviation for effectiveness scores and 0.26 standard deviation for PVAAS scores). 

Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


Research question 2: What proportions of teachers leave their school and leave the district, both 
during the school year and between school years? 

The main report presents findings on teacher turnover in each school year and by teacher and school 
characteristics for characteristics that had differences between groups of at least 5 percentage points and that 
had the largest relationship with teacher turnover based on the analyses for research question 3. This section 
presents teacher turnover rates by teacher and school characteristics by school year and for additional teacher 
and school characteristics not included in the main report. 


Teacher turnover by school year. Table B7 presents the percentage of teachers who left their school during the 
full year, within each school year, and between school years. The percentage of teachers leaving during the full 
year remained consistent at around 20-22 percent across school years, except in 2010/11, during the districtwide 
initiative that converted public schools to charter schools, and in 2012/13, during the budget shortfall (as 
discussed in the main report). In those two years the percentage of teachers leaving increased. Teachers left 
primarily between school years; the percentage of teachers leaving within the school year did not change 
substantially over the school years in the study. 


Table B7 also presents the percentage of teachers who left their school but remained in the district during each 
school year and the percentage who left the district. A higher percentage of teachers changed schools within the 
district during each year than left the district. Again, there were increases in both teachers changing schools and 
teachers leaving the district in 2010/11 and 2012/13. 


Table B7. Teacher turnover in the School District of Philadelphia, by school year, 2010/11-2016/17 (percent of 
teachers leaving their school or the district) 


Leaving their 


Leaving during Leaving within Leaving between school but not Leaving the 
Number of the full year the school year school years the district district 
NYel ite) BY(-t- 10 teachers (Oct. to Oct.) (Oct. to Apr.) (Apr. to Oct.) during full year during full year 
2010/11 10,054 32.8 3.2 30.9 21.9 11.3 
2011/12 9,038 21.3 2.4 19.5 14.8 6.6 
2012/13 8,824 31.6 2.2 30.3 21.3 10.4 
2013/14 8,137 22.6 2.6 20.9 15.5 7.2 
2014/15 8,061 20.3 3.2 17.9 13.6 6.8 
2015/16 8,015 22.9 3.0 20.6 15.7 7.2 
2016/17 8,139 20.3 3.2 18.0 12.9 7.5 
All school years 60,268 24.8 2.8 22.9 16.8 8.2 
2014/15-2016/17 24,215 21.2 3.1 18.8 14.1 7.2 


Note: The percentage of teachers leaving within the school year and the percentage leaving between school years do not sum to the percentage leaving 
during the full year because some teachers might have left their school within the school year and then changed schools again between school years. In 
these cases the teacher is included in both the within the school year and between school years columns but is counted only once in the full year column. 
Similarly, the percentage of teachers leaving their school but not the district and the percentage of teachers leaving the district do not always sum to the 
total percentage leaving during the full year because some teachers might have changed schools during the school year and also left the district during the 
same school year. In these cases the teacher is included in both the percentage of teachers leaving their school but not the district and in the percentage of 
teachers leaving the district columns but is counted only once in the full year column. 

Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11-2016/17. 


Teacher turnover by teacher characteristic and school year. Table B8 presents teacher turnover by teacher 
characteristics and by school year for the characteristics discussed in the main report. It also includes turnover for 
other teacher characteristics that did not have differences between groups of teachers of at least 5 percentage 
points or that did not have a sufficiently large relationship with teacher turnover based on the analyses conducted 
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for research question 3 to be included in the main report. Table B9 presents the percentage of teachers leaving 
the district by all teacher characteristics and by school year. Tables B10 and B11 show the percentages of teachers 
leaving their school and leaving the district by teacher evaluation rating and by school year. 


Table B8. Percentage of teachers leaving their school in the School District of Philadelphia, by teacher 
characteristic and school year, 2010/11-2016/17 


(o)¥{-) ¢-11) 


Percent Sample 


Teacher characteristic leaving size? 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 
Overall 24.8 60,268 32.8 21.3 31.6 22.6 20.3 22.9 20.3 
Years teaching in the district u ; iu u u i 4 t 
Four or fewer years 37.2 15,372 47.9 31.0 43.5 31.3 29.7 33.2 31.0 
5-10 years 19.9 16,369 19.5 16.4 22.3 21.0 19.6 23.6 16.5 
11-17 years 16.7 14,501 18.4 13.3 21.9 17.5 14.3 16.8 15.3 
More than 17 years 21.5 12,595 28.6 18.0 33.3 17.8 17.5 17.7 15.8 
Race/ethnicity q u 

Black 24.6 15,002 30.6 20.1 32.4 22.6 19.0 23.6 21.7 
Hispanic 25.1 1,703 32.7 22.8 28.4 25.0 20.7 25.8 19.5 
White 23.6 40,078 31.8 20.6 30.1 21.1 19.3 21.4 18.9 
Other race/ethnicity 25.2 2,031 38.1 20.6 28.6 24.1 20.7 18.9 22.1 
Gender u “ 

Female 22.8 43,468 29.2 19.4 28.9 21.3 18.6 21.4 19.4 
Male 27.3 15,370 38.4 23.6 35.6 22.7 21.6 23.9 21.1 
Subject taught, grades 6-8 

Math/science 28.2 5,724 25.7 27.8 35.6 28.3 25.5 29.5 24.0 


English language arts/social studies 27.8 6,006 26.1 25.6 34.7 26.7 26.4 30.8 24.5 


Subject taught, grades 9-12 ¥ 


Math/science 28.3 5,169 39.4 26.9 40.5 25.8 20.3 20.3 19.2 


English language arts/social studies 26.7 5,903 39.1 24.3 36.5 21.1 20.8 21.3 18.1 


General education/special education 


General education 23.6 48,779 31.0 20.4 30.6 21.1 19.0 21.4 19.6 


Special education 25.7 10,059 35.3 20.8 30.8 24.3 20.8 25.1 20.8 


Grade level taught 


Elementary grades (kindergarten—5) 21.8 34,193 27.0 18.0 27.0 20.2 18.6 21.4 19.5 


Middle school grades (6-8) 28.5 19,825 33.3 25.0 34.3 27.5 24.3 28.8 24.5 


High school grades (9-12) 27.8 16,539 42.2 23.7 37.9 23.5 20.2 20.5 18.8 
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Overall 


Percent Sample 


Teacher characteristic leaving size? 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 
Teacher attendance ¢ u u u u ui f t 
Regular (95 percent attendance 18.9 35,426 26.5 15.4 26.4 15.9 13.6 16.3 14.8 
or more) 


Frequently absent (75-94 percent 29.0 20,596 38.7 25.3 34.4 27.1 24.2 27.6 24.5 
attendance) 


Less than 75 percent attendance 51.7 2,794 58.1 50.5 55.2 49.4 51.8 48.9 46.1 
Commute time } z if f : ‘i i 

Shorter than 15 minutes 18.6 15,037 24.5 14.8 21.1 18.8 16.3 17.8 16.5 
15-29 minutes 21.7 25,499 27.7 16.4 25.9 21.9 18.7 20.2 20.5 
30-59 minutes 22.9 13,042 28.4 18.1 26.8 22.6 21.0 22.7 20.5 
60 minutes or longer 36.7 968 48.6 23.6 35.9 36.2 33.6 35.0 37.7 


+ Turnover rates differ by at least 5 percentage points between groups within the category. 

Note: The total number of teachers by grade level taught sums to more than the total sample size because teachers may teach at multiple grade levels. 
Teachers may also teach in multiple subject areas. The total sample size for each category may not sum to the overall sample size because some teachers 
are missing data for some characteristics and are excluded from the turnover calculation. 

a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district from 2010/11 through 2016/17). 

Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11—2016/17. 


Table B9. Percentage of teachers leaving the School District of Philadelphia, by teacher characteristic and 
school year, 2010/11-2016/17 


(o)Y{=) ¢-11) 


Percent Sample 


Teacher characteristic leaving size? 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 
Overall 8.2 60,268 11.3 6.6 10.4 7.2 6.8 7.2 75 
Years of teaching in the district a u ‘ u i v f u 
Four or fewer years 11.0 15,372 14.9 8.5 9.2 9.7 9.8 10.8 11.2 
5-10 years 5.2 16,369 3.7 3.7 5.3 6.0 5.8 6.6 5.0 
11-17 years 4.4 14,501 4.4 3.2 5.8 4.7 4.0 4.4 4.1 
More than 17 years 13.1 12,595 18.8 10.3 23.6 9.4 9.8 8.5 9.2 
Race/ethnicity 

Black 7.8 15,002 9.5 5.9 10.8 7.8 6.3 7.1 6.6 
Hispanic 7.0 1,703 12.0 5.5 6.9 5.8 6.6 6.3 5.7 
White 8.4 40,078 11.6 6.8 10.6 7.0 7.0 7.1 7.6 
Other race/ethnicity 8.3 2,031 13.8 5.6 7.8 6.6 7.9 7.4 8.0 
Gender 

Female 8.1 43,468 10.7 6.3 10.6 7.5 6.8 6.9 7.2 
Male 8.4 15,370 12.2 6.9 9.9 6.1 6.7 7.9 7.8 
Subject taught, grades 6-8 

Math/science 7.7 5,724 75 6.8 10.2 7.1 8.0 7.3 6.9 


English language arts/social studies 7.8 6,006 7.9 5.8 9.7 5.9 7.9 10.2 7.8 


(o)Y{-) ¢-11) 


Percent Sample 
Teacher characteristic leaving size? 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 


Subject taught, grades 9-12 


Math/science 8.9 5,169 9.2 8.8 12.5 9.3 7.2 6.8 7.6 
English language arts/social studies 8.1 5,903 10.3 7.7 10.8 6.0 6.6 7.2 7.0 
General education/special education 

General education 7.9 48,779 11.1 6.3 9.7 6.8 6.6 6.9 7.2 
Special education 9.4 10,059 11.2 7.4 14.0 8.8 7.6 8.3 7.9 
Grade level taught 

Elementary grade (kindergarten—5) 75 34,193 10.0 5.7 9.7 7.1 6.4 6.5 7.0 
Middle school grade (6-8) 9.2 19,825 12.3 6.6 11.4 8.4 7.9 8.7 8.7 
High school grade (9-12) 9.4 16,539 13.9 8.0 11.8 7.6 7.0 7.5 7.7 
Teacher attendance u u i i i i x 
Regular (95 percent attendance 5.5 35,426 8.2 3.9 8.3 4.5 3.7 4.1 4.7 
or more) 

Frequently absent (75-94 percent 10.6 20,596 14.9 8.7 11.9 9.4 9.0 9.9 9.9 
attendance) 

Less than 75 percent attendance 23.7 2,794 27.3 22.8 26.5 21.5 25.1 22.1 19.8 
Commute time iH i f a i i 
Shorter than 15 minutes 7.1 15,037 9.8 6.2 8.6 6.4 5.9 6.1 6.3 
15-29 minutes 8.1 25,499 10.7 6.5 11.2 6.9 6.7 6.7 7.6 
30-59 minutes 8.3 13,042 10.6 5.8 10.6 7.7 7.6 8.1 7.4 
60 minutes or longer 15.4 968 21.6 9.8 9.9 16.9 12.5 20.0 14.6 


+ Turnover rates differ by at least 5 percentage points between groups within the category. 

Note: The total number of teachers by grade level taught sums to more than the total sample size because teachers may teach at multiple grade levels. 
Teachers may also teach in multiple subject areas. The total sample size for each category may not sum to the overall sample size because some teachers 
are missing data for some characteristics and are excluded from the turnover calculation. 

a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district from 2010/11 through 2016/17). 

Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11-2016/17. 


Table B10. Percentage of teachers leaving their school in the School District of Pennsylvania, by teacher 
evaluation rating and school year, 2014/15-2016/17 


Overall 
Percent 

Teacher evaluation rating leaving 2014/15 2015/16 2016/17 
Overall 21.2 24,215 20.3 22.9 20.3 
Teacher effectiveness score u i i y 
Failing or needs improvement (0—-1.49)? 39.7 914 39.6 49.5 33.3 
Proficient (1.5-2.49) 17.1 18,564 15.8 20.7 15.0 
Distinguished (2.5-3) 11.0 2,629 7.2 12.7 9.5 
Pennsylvania Value-Added Assessment System score a 2 u A 
Failing or needs improvement (0-1.49) 23.5 745 27.4 42.9 14.9 
Proficient (1.5-2.49) 16.6 2,967 15.5 19.8 10.3 
Distinguished (2.5-3) 15.2 1,540 12.1 18.5 5.0 


+ Turnover rates differ by at least 5 percentage points. 

Note: The total sample size for each category does not sum to the overall sample size because some teachers are missing data for an evaluation rating and 
are excluded from the turnover calculation. 

a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district from 2014/15 through 2016/17). 

b. The bottom two evaluation rating categories (failing and needs improvement) were combined because of small sample sizes in the failing category. 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15-2016/17. 


Table B11. Percentage of teachers leaving the School District of Philadelphia, by teacher evaluation rating and 
school year, 2014/15-2016/17 


Overall 

Percent Sample 
Teacher evaluation rating leaving size? 2014/15 2015/16 2016/17 
Overall 7.2 24,215 6.8 7.2 75 
Teacher effectiveness score u 1 ‘ 
Failing or needs improvement (0—1.49)? 12.0 914 9.7 14.4 12.9 
Proficient (1.5-2.49) 4.8 18,564 4.4 4.9 5.1 
Distinguished (2.5-3) 3.5 2,629 2.8 3.4 4.1 
Pennsylvania Value-Added Assessment System score x 
Failing or needs improvement (0—-1.49) 6.7 745 6.0 13.0 4.7 
Proficient (1.5—-2.49) 3.9 2,967 3.9 3.8 4.0 
Distinguished (2.5—3) 3.9 1,540 3.2 4.4 3.1 


+ Turnover rates differ by at least 5 percentage points. 

Note: The total sample size for each category does not sum to the overall sample size because some teachers are missing data for an evaluation rating and 
are excluded from the turnover calculation. 

a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district from 2014/15 through 2016/17). 

b. The bottom two evaluation rating categories (failing and needs improvement) were combined because of small sample sizes in the failing category. 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


Teacher turnover by school characteristic and school year. The main report describes differences in the schools 
that teachers left and schools where teachers stayed, highlighting school characteristics with a difference of at 
least 5 percentage points and a large relationship with teacher turnover based on the analyses for research 
question 3. This section presents the results by other school characteristics and by school year (table B12). This 
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section also presents the results of an analysis of differences between the schools where teachers who left the 
district taught and schools where teachers who stayed in the district taught (table B13). 


Table B12. Characteristics of the schools in which teachers taught in the School District of Philadelphia, by 
whether teachers left or stayed in their school and by school year, 2014/15-2016/17 


Overall 2014/15 2015/16 2016/17 


Stayed Stayed Stayed Stayed 


Left in Sample Left in Left in Left in 
AYol Yolo) olat-Je-leic-la kd (a school school size? To} YoYo) mol see) INN ol solo) NNN ol p00) INN 0] soe) MNNEEECYol poo) | 


Teacher survey construct score (percent positive responses) 


School leadership 74.9 78.8 22,206 73.3 77.3 75.1 79.7 76.2 79.3 
School climate 51.5 56.8 22,206 51.3 55.8 51.0 57.0° 52.3 57.5* 
Teacher autonomy 88.0 89.2 22,206 88.0 89.2 88.5 89.6 87.4 88.7 
Teacher respect 70.6 72.0 22,206 71.5 72.5 70.8 72.7 69.4 70.8 
Student behavior 81.6 83.8 22,206 81.4 83.7 81.4 83.9 82.2 83.9 
Collaboration 36.2 36.5 22,206 33.7 33.7 32.1 32.8 43.2 43.2 


School student characteristics (percent of students) 
Advanced or proficient in math 15.9 22.9* 23,976 15.6 22.0° 16.0 24.47 16.2 22.57 


Advanced or proficient in English 28.9 36.5' 23,976 28.6 35.8" 28.9 37.6" 29.2 36.0* 
language arts 


Economically disadvantaged students 96.3 93.3 24,204 95.3 92.1 96.3 93.0 97.5 94.8 


Black students 59.3 50.4 24,205 59.5 51.4" 58.0 50.2* 60.4 49.6" 
Hispanic students 20.1 20.0 24,199 20.2 19.6 20.6 19.9 19.6 20.5 
White students 9.5 14.2 24,199 9.7 14.2 9.9 13.8 8.9 14.5 


Principal leadership 


Years principal led the school 2.9 3.2 24,205 2.7 2.9 2.7 3.1 3.2 3.5 


+ Characteristics of schools that teachers left are at least 5 percentage points different from characteristics of schools where teachers remained . 
a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district from 2014/15 through 2016/17). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 
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Table B13. Characteristics of the schools in which teachers taught, by whether teachers left or stayed in the 
School District of Philadelphia and by school year, 2014/15—2016/17 


Overall 2014/15 2015/16 2016/17 


Stayed Stayed Stayed Stayed 


Left Ta) Sample Left in Left in Left in 
AYol solo) olat-Je- leita kia (a district district size? district district district district district district 


Teacher survey construct score (percent positive responses) 


School leadership 76.2 78.1 22,206 74.9 76.6 76.8 78.8 76.8 78.8 
School climate 53.5 55.8 22,206 52.9 55.0 53.7 55.8 53.8 56.7 
Teacher autonomy 88.5 89.0 22,206 88.2 89.0 89.3 89.4 87.9 88.4 
Teacher respect 71.0 71.8 22,206 71.7 72.4 71.8 72.3 69.8 70.6 
Student behavior 82.6 83.4 22,206 82.4 83.2 82.6 83.4 82.6 83.6 
Collaboration 36.8 36.4 22,206 33.9 33.6 32.9 32.6 43.1 43.2 


School student characteristics (percent of students) 


Advanced or proficient in math 19.0 21.6 23,976 18.7 20.8 19.4 22.7 18.7 21.4 


Advanced or proficient in English 


32.2 35.1 23,976 31.8 34.5 32.8 35.8 32.0 34.9 
language arts 


Economically disadvantaged students 95.2 93.9 24,204 93.3 92.7 95.6 93.6 96.4 95.3 


Black students 57.0 51.9t 24,205 56.5 52.8 55.9 51.7 58.4 51.37 
Hispanic students 19.1 20.1 24,199 18.8 19.8 19.6 20.1 18.7 20.4 
White students 11.2 13.3 24,199 12.3 13.3 10.9 13.1 10.6 13.6 


School leadership 


Years principal led the school 2.9 3.1 24,205 2.6 2.9 2.8 3.0 3.3 3.4 


+ Characteristics of the schools that teachers left are at least 5 percentage points different from characteristics of the schools where teachers remained. 
a. The number of teacher-years in the analysis (teachers multiplied by years employed in the district from 2014/15 through 2016/17). 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


Research question 3: How does teaching experience relate to whether teachers leave their school or 
leave the district? 

In addition to examining annual teacher turnover rates, the study team calculated the rate at which teachers left 
their school or left the district for each year of teaching in their school or in the district and the cumulative 
probability that teachers would leave their school or leave the district over time. The main report provides figures 
showing the percentage of teachers leaving for each year of teaching and the cumulative probability that teachers 
would leave their school or district. Table B14 summarizes the percentage of teachers leaving and the cumulative 
probability for each of the first 5 years of teaching and every 5 years between 5 and 30 years of teaching. 
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Table B14. Teacher turnover in the School District of Philadelphia, by years of teaching in their school or 
district, 2010/11—-2016/17 


By years teaching in their school By years teaching in the district 
Cumulative Cumulative 
Percentage leaving _ probability of leaving | Percentageleaving probability of leaving 
Years of teaching their school their school the district the district 
1 46 38 17 15 
2 33 56 15 28 
3 24 65 11 35 
4 22 72 7 40 
5 20 77 10 45 
10 14 90 4 59 
15 14 96 4 67 
20 12 98 6 75 
25 19 99 11 84 
30 23 100 17 92 


Note: The percent leaving and cumulative probability are not identical in year 1 because the percent leaving averages the percent of teachers who left across 
the three time points observed in year 1 (September, October, and April) while the cumulative probabilty summarizes the likelihood teachers would leave 
by the end of their first year. 

Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11-2016/17. 


Research question 3a: What other teacher and school characteristics are related to teachers leaving 
their school or leaving the district? 

In addition to looking at differences in teacher turnover separately by teacher and school characteristics, the study 
team examined how teacher and school characteristics influenced teacher turnover, after other characteristics 
were taken into account. The study team examined which teacher and school characteristics were related to 
teachers leaving their school and teachers leaving the district. Appendix A describes the methods used for each 
analysis. Only teachers with complete data across all years were included in the hazards models, which limited 
the sample to school years 2014/15 through 2016/17. 


The main report discusses the teacher and school characteristics with the largest relationship to the likelihood of 
teachers leaving their school. Tables B15 and B16 provide detailed results for these characteristics, as well as 
results for additional teacher and school characteristics and results from the model of characteristics related to a 
teacher’s likelihood of leaving the district. Table B17 provides results from an alternative model of teachers leaving 
the district that included time-dependent effects. 


Results of the hazards model analysis of teachers leaving their school and leaving the district. The coefficient for 
each variable in table B15 describes the effect of a one-unit difference in that variable on the log of the probability 
of teachers leaving their school or leaving the district. Coefficients from a Cox proportional hazards model can be 
difficult to interpret; therefore, table B15 also shows the hazard ratio and its confidence interval. The hazard ratio 
is the antilog of the coefficient (e'°*“") and describes the effect of a one-unit difference in each variable on the 
probability of a teacher leaving. For example, the hazard ratio for retirement eligibility tells us that teachers who 
are eligible for retirement are 370 percent more likely to leave the district than teachers who are not eligible for 
retirement (calculated as the hazard ratio multiplied by 100). In contrast, teachers who are vested in the pension 
system (teachers who have worked enough years to qualify for retirement benefits) are 97 percent less likely to 
leave their district than teachers who are not vested (calculated by subtracting 1 from the hazard ratio and 
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multiplying by 100). The hazard ratio is expressed in terms of the percentage change in a teacher’s probability of 
leaving. 


Interpreting the influence of a characteristic on teacher turnover using hazard ratios when a one-unit change is 
very small can be difficult. For example, commute time is measured in minutes. A one-minute increase in commute 
time increases a teacher’s probability of leaving the district by 1 percent. It might be more useful, though, to know 
the effect of a 10-minute increase in commute time, which can be calculated with the formula e!orfficient*unit change) 
For this example, a 10-minute increase in commute time increases a teacher’s probability of leaving the district 
by 7.7 percent (e!207"20). 


Table B16 presents the change in teachers’ likelihood of leaving their school or the district with different 
magnitude changes in each characteristic that does not represent a teacher subgroup. Subgroups are excluded 
because a one-unit change represents belonging to that subgroup and therefore the hazard ratio provided in table 
B15 describes the effect of belonging to the group on teacher turnover. 


Table B15. Coefficients and hazard ratios from discrete time hazards model of teachers leaving their school 
and leaving the School District of Philadelphia, 2014/15-2016/17 


Leaving their school Leaving the district 


Coefficient 95% Coefficient 95% 


(Genrecrce| Hazard confidence (standard Hazard confidence 
NWETat-) 9) (=) error) elite) interval error) ratio interval 


Teacher characteristic 


Eligible for retirement 0.555** 1.74 (1.53, 1.308** 3.70 (3.05, 
(0.065) 1.98) (0.097) 4.48) 
Vested in pension system —0.417** 0.66 (0.59, —3.561** 0.03 (0.01, 
(0.055) 0.73) (0.423) 0.07) 
Number of previous school moves 0.095** 1.10 (1.05, na na na 
(0.021) 1.15) 
Female —0.036 0.96 (0.89, —0.049 0.95 (0.82, 
(0.041) 1.05) (0.078) 1.11) 
Black 0.262** 1.30 (1.07, 0.275 1.32 (0.94, 
(0.100) 1.58) (0.171) 1.84) 
Special education teacher —0.086 0.92 (0.83, —0.102 0.90 (0.76, 
(0.048) 1.01) (0.088) 1.07) 
Frequently absent 0.401** 1.49 (1.39, 0.652** 1.92 (1.68, 
(0.037) 1.60) (0.069) 2.20) 
Less than 75 percent attendance 0.758** 2.13 (1.88, 1.057** 2.88 (2.28, 
(0.065) 2.42) (0.119) 3.63) 
Teacher effectiveness score —0.699** 0.50 (0.45, —0.674** 0.51 (0.42, 
(0.049) 0.55) (0.094) 0.61) 
Commute time between home and school 0.005** 1.00 (1.00, 0.007** 1.01 (1.00, 
(0.001) 1.01) (0.002) 1.01) 
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School characteristic 


Leaving their school 


Coefficient 


Geneelce| 
error) 


Hazard 
ratio 


95% 


Leaving the district 


Coefficient 
confidence (standard 
interval 


error) 


Hazard 
ratio 


95% 


confidence 
interval 


Grades 6-8 0.161** 1.18 (1.08, 0.054 1.05 (0.91, 
(0.041) 1.27) (0.076) 1.22) 
Grades 9-12 -0.070 0.93 (0.83, 0.133 0.88 (0.71, 
(0.060) 1.05) (0.105) 1.08) 
Percent of Black students (among non-Black 0.003** 1.00 (1.00, 0.004* 1.00 (1.00, 
teachers) (0.001) 1.00) (0.001) 1.01) 
Percent of Black students (among Black teachers) —0.004** 1.00 (0.99, —0.008** 0.99 (0.99, 
(0.001) 1.00) (0.003) 1.00) 
Percent of students scoring proficient or advanced —0.004* 1.00 (0.99, 0.000 1.00 (0.99, 
in math (0.002) 1.00) (0.003) 1.01) 
Years principal led the school —0.021** 0.98 (0.97, -0.017 0.98 (0.96, 
(0.007) 0.99) (0.013) 1.01) 
Student—teacher ratio 0.007* 1.01 (1.00, 0.019** 1.02 (1.01, 
(0.003) 1.01) (0.005) 1.03) 
Teacher survey construct 
School leadership —0.003* 1.00 (0.99, —0.006* 0.99 (0.99, 
(0.002) 1.00) (0.003) 1.00) 
School climate —0.016** 0.98 (0.98, —0.003 1.00 (0.99, 
(0.003) 0.99) (0.005) 1.01) 
Teacher autonomy —0.003 1.00 (0.99, 0.002 1.00 (0.99, 
(0.004) 1.00) (0.007) 1.02) 
Teacher respect 0.005 1.00 (1.00, 0.010 1.01 (0.99, 
(0.004) 1.01) (0.009) 1.03) 
Student behavior 0.002 1.00 (0.99, 0.006 1.01 (0.99, 
(0.004) 1.01) (0.008) 1.02) 
Collaboration -0.001 1.00 (1.00, 0.000 1.00 (0.99, 
(0.002) 1.00) (0.003) 1.01) 
Number of teacher observations 48,052 50,147 


* Significant at p <.05; ** significant at p <.01. 
na is not applicable (not included in the model). 


Source: Authors’ analysis of data from the School District of Philadelphia for 2010/11-2016/17. 


Table B16. Marginal effects of changes in teacher and school characteristics on teachers’ probability of leaving 
their school or the School District of Philadelphia, 2014/15-2016/17 (percentage change) 


Teacher or school characteristic Effect on leaving their school Effect on leaving the district 


Number of previous school moves 


One prior move +9.9 na 


Two prior moves +20.9 na 


Teacher effectiveness score 


1/3-point increase —20.6 -19.9 


1/2-point increase -29.5 28.6 


Commute time between home and school 


10-minute increase +5.5 +7.7 


20-minute increase +11.3 +15.9 


Percent of Black students (among non-Black teachers) 


10 percentage point increase +3.0 +3.8 


20 percentage point increase +6.1 +7.7 


Percentage of Black students (among Black teachers) 


10 percentage point increase -1.4 —4.3 


20 percentage point increase —2.8 -8.5 


Percent of students scoring proficient or advanced in math 


10 percentage point increase —4.0 -0.1 


20 percentage point increase -7.9 —0.2 


Years principal led the school 


Two-year increase —4.1 3.3 


Three-year increase -6.1 -4.9 


Student—teacher ratio 


5-student increase $3.5 +9.8 


10-student increase +7.1 +20.5 


School leadership (survey construct) 


10 percentage point increase in positive responses 3.3 -6.0 


20 percentage point increase in positive responses -6.5 —11.7 


School climate (survey construct) 


10 percentage point increase in positive responses —14.7 —2.5 


20 percentage point increase in positive responses —27.2 —5.0 


Teacher autonomy (survey construct) 


10 percentage point increase in positive responses —2.5 +1.6 


20 percentage point increase in positive responses —4.9 +3.3 


Teacher respect (survey construct) 


10 percentage point increase in positive responses +4.8 +10.4 


20 percentage point increase in positive responses +9.9 +21.8 
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Teacher or school characteristic Effect on leaving their school Effect on leaving the district 


Student behavior (survey construct) 


10 percentage point increase in positive responses +2.4 +6.7 


20 percentage point increase in positive responses +4.8 +13.9 


Collaboration (survey construct) 


10 percentage point increase in positive responses -0.6 +0.5 


20 percentage point increase in positive responses -1.3 +1.0 


na is not applicable (not included in the model). 
Source: Authors’ analysis of data from the School District of Philadelphia. 


Results from the hazards model analysis of teachers leaving the district with time-dependent effects. For each 
hazards model described above (the model of teachers leaving their school and the model of teachers leaving the 
district), the study team tested the proportional hazards assumption. As described in appendix A, the proportional 
hazards assumption assumes that the change in risk with each covariate is constant over time. This assumption 
might be violated if, for example, a variable increases a teacher’s risk of leaving in the first five years of teaching 
but has no effect on turnover later in a teacher’s tenure. 


For the model of teachers leaving the district, one variable, retirement eligibility, violated the proportional hazards 
assumption, meaning that its influence on teachers’ likelihood of leaving changed over time. This is to be expected 
since teachers typically become eligible for retirement near the end of their career; retirement eligibility would 
thus have no effect early in a teacher’s career but would have a potentially large effect in later years. To adjust 
for this, the study team estimated a model interacting retirement eligibility with time. Table B17 presents the 
results for this model. Results for the model with time-dependent effects are similar to results of the model 
without time-dependent effects. The same characteristics are significantly related to the likelihood of teachers 
leaving the district in both models. Coefficients in both models did not differ for 8 of the characteristics and were 
similar for the remaining 13 characteristics. 


For teacher characteristics other than retirement eligibility, the results in table B17 can be interpreted in the same 
way as the results in table B15. The results for retirement eligibility can be interpreted as follows: In their first year 
of teaching in the district, teachers who are eligible for retirement are 10 times more likely to leave than teachers 
who are not eligible for retirement. The effect of retirement eligibility then decreases by 5 percent for each 
subsequent year. 


Because very few teachers would be eligible for retirement in their first year employed in the district, it makes 
more sense to look at the effect of retirement at 30 years, when most teachers are eligible to retire. At 30 years 
of employment in the district, teachers who are eligible for retirement are two times as likely to leave as teachers 
who are not eligible for retirement.” This is less than the 3.7 times higher risk of leaving reported for teachers 
eligible for retirement in the model without time-dependent effects. This difference is due to differences in how 
the change in risk is calculated in each model. The model without time-dependent effects presents the average 
increase in risk across the length of a teacher’s career in the district rather than the increase at one time point 
(such as 30 years). 


2 This is calculated as 10.61*(e*In(0.95)*30). 
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Table B17. Coefficients and hazard ratios from discrete time hazards model of teachers leaving the School 
District of Philadelphia, with time-dependent effects, 2014/15-2016/17 


Coefficient Hazard 95% 


WETar]e)(-1 (standard error) ratio confidence interval 


Teacher characteristic 


Vested in pension system —3.652** 0.03 (0.01, 0.06) 
(0.419) 

Female —0.043 0.96 (0.82, 1.11) 
(0.077) 

Black 0.257 1.29 (0.93, 1.80) 
(0.125) 

Special education teacher —0.126 0.88 (0.74, 1.05) 
(0.088) 

Frequently absent 0.658** 1.93 (1.69, 2.21) 
(0.069) 

Less than 75 percent attendance 1.057** 2.88 (2.28, 3.63) 
(0.119) 

Teacher effectiveness score —0.667** 0.51 (0.43, 0.62) 
(0.093) 

Commute time between home and school 0.008** 1.01 (1.00, 1.01) 
(0.002) 


School characteristic 


Grades 6-8 0.060 1.06 (0.92, 1.23) 
(0.076) 

Grades 9-12 —0.131 0.88 (0.72, 1.08) 
(0.104) 

Percent of Black students (among non-Black teachers) 0.003* 1.00 (1.00, 1.01) 
(0.001) 

Percent of Black students (among Black teachers) —0.008** 0.99 (0.99, 1.00) 
(0.003) 

Percent of students scoring proficient or advanced in math 0.000 1.00 (0.99, 1.01) 
(0.003) 

Years principal led the school —0.017 0.98 (0.96, 1.01) 
(0.013) 

Student—teacher ratio 0.019** 1.02 (1.01, 1.03) 
(0.005) 
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Coefficient Hazard 95% 


WETar])(-1 (standard error) ratio confidence interval 

Teacher survey construct 

School leadership —0.006 0.99 (0.99, 1.00) 
(0.003) 

School climate —0.002 1.00 (0.99, 1.01) 
(0.005) 

Teacher autonomy 0.002 1.00 (0.99, 1.02) 
(0.007) 

Teacher respect 0.009 1.01 (0.99, 1.03) 
(0.009) 

Student behavior 0.006 1.01 (0.99, 1.02) 
(0.008) 

Collaboration 0.001 1.00 (0.99, 1.01) 
(0.003) 

Eligible for retirement: time-dependent effects 

First year 2.362** 10.61 (7.05, 15.99) 
(0.209) 

Change in effect with each year —0.548** 0.95 (0.93, 0.97) 
(0.010) 

Number of teacher observations 50,147 


* Significant at p <.05; ** significant at p <.01. 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 


Research question 3b: Does the relationship between teacher and school characteristics and turnover 
differ between effective teachers and ineffective teachers? 

Table B18 presents results for the analysis of whether the relationship between teacher and school characteristics 
and teacher turnover differed between teachers rated proficient or distinguished (effective teachers) and teachers 
rated failing or needs improvement (ineffective teachers). Tables B18 and B19 compare findings for the two 
groups of teachers on all variables identified under research question 3a as substantively related to turnover and 
policy relevant. One characteristic, school climate, had a significantly different relationship with turnover for 
teachers rated proficient or distinguished than for teachers rated failing or needs improvement. In addition to 
being statistically significant, school climate had a substantively different relationship with turnover depending on 
teachers’ ratings. As described in the main report, a 10 percentage point increase in positive responses to school 
climate survey questions was related to a 5 percent increase in the likelihood that teachers rated failing or needs 
improvement would leave their school. However, for teachers rated proficient or distinguished, a 10 percentage 
point increase in positive responses to school climate survey questions was related to a 18 percent decrease in 
the likelihood of leaving their school (for example, from 20 percent to 16 percent). Table B19 presents the change 
in teachers’ likelihood of leaving their school for each group of teachers with different magnitude changes in 
characteristics other than teacher subgroups. 
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Table B18. Coefficients and hazard ratios for interactions between teachers’ evaluation ratings and other 
teacher and school characteristics for model of teachers leaving their school in the School District of 
Philadelphia, 2014/15-2016/17 


Teachers leaving their school 


Coefficient Hazard 95% 
WETar]e)(-1 (standard error) ratio confidence interval 


Number of previous school moves 


Teachers rated failing or needs improvement 0.046 1.05 (0.97, 1.13) 
(0.039) 

Teachers rated proficient or distinguished 0.058 1.06 (0.98, 1.15) 
(0.042) 

Identifying as Black 

Teachers rated failing or needs improvement —0.023 0.98 (0.53, 1.81) 
(0.314) 

Teachers rated proficient or distinguished 0.269 1.31 (0.68, 2.52) 
(0.335) 

Frequently absent 

Teachers rated failing or needs improvement 0.303 1.35 (1.06, 1.73) 
(0.124) 

Teachers rated proficient or distinguished 0.126 1.13 (0.88, 1.47) 
(0.131) 

Grades 6-8 

Teachers rated failing or needs improvement 0.028 1.03 (0.78, 1.36) 
(0.141) 

Teachers rated proficient or distinguished 0.154 1.17 (0.87, 1.56) 
(0.149) 


School climate (survey construct) 


Teachers rated failing or needs improvement 0.005 1.00 (0.99, 1.02) 
(0.009) 

Teachers rated proficient or distinguished —0.025* 0.98 (0.96, 0.99) 
(0.009) 


* Differences between groups are significant at p <.05. 
Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15—2016/17. 
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Table B19. Marginal effects of changes in teacher and school characteristics on teachers’ probability of leaving 
their school in the School District of Philadelphia for interactions between teachers’ evaluation ratings and 
other teacher and school characteristics, 2014/15-2016/17 (percentage change) 


Teachers rated Teachers rated 
proficient or distinguished 


Teacher and school characteristic failing or needs improvement 


Number of previous school moves 


One prior move +4.7 +11.0 


Two prior moves +9.7 +23.3 


School climate (survey construct) 


10 percentage point increase in positive responses +5.0 -17.9 


20 percentage point increase in positive responses +10.2 —32.6 


Source: Authors’ analysis of data from the School District of Philadelphia for 2014/15-2016/17. 


